Quality Assurance Engineer

Accountabilities

• Build, maintain, and expand automated testing for AI-powered voice and chat agents, covering individual conversational turns through complete roleplay and end-to-end flows.

• Design and execute tests for LLM output quality, including correctness, consistency, structured outputs, language fidelity, prompt regressions, and other non-deterministic behaviors.

• Develop evaluation approaches using LLM-as-a-judge techniques, golden datasets, tolerance-based assertions, and other AI evaluation methodologies.

• Validate RAG pipelines, with particular attention to retrieval relevance, grounding, faithfulness, and overall answer quality.

• Test voice pipelines, including speech-to-text and text-to-speech accuracy, multilingual behavior, and real-time, low-latency performance.

• Automate testing across UI/E2E, API, database, AI-output, desktop, and mobile layers.

• Build and improve CI/CD quality processes, including linting, type checks, automated tests, AI evaluations, coverage gates, and deployment checks.

• Own release-quality activities, including regression strategies, early identification of defects, and evidence-based go/no-go recommendations.

• Take ownership of existing QA automation suites, improve their coverage and reliability, and establish stronger shared frameworks, tooling, processes, and QA standards.

• Support load and performance testing and continuously adapt quality practices as products, priorities, and technical challenges evolve.

Requirements

• 5+ years of experience as a QA Automation Engineer, including proven hands-on experience testing AI-powered systems rather than only traditional software.

• Demonstrated ability to build testing frameworks, QA standards, and CI/CD-integrated automation from the ground up.

• Practical experience testing conversational AI, including voice and/or chat agents, in real-world production projects.

• Strong understanding of LLMs, prompt engineering, RAG, and approaches for testing and evaluating non-deterministic AI outputs.

• Knowledge of voice technologies and pipelines, including speech-to-text and text-to-speech, and experience automating their testing.

• Hands-on experience with LLM evaluation or observability tools such as Langfuse, LangSmith, DeepEval, or RAGAS.

• Strong proficiency in UI/E2E and API test automation, including tools such as Playwright, as well as experience creating or significantly improving CI/CD pipelines.

• Experience with database testing, load and performance testing, and desktop or mobile test automation.

• Strong Git practices, basic cloud knowledge, and familiarity with Agile development and sprint-based delivery.

• A pragmatic, hands-on approach and the ability to work effectively in fast-paced, rapidly changing environments.

• Strong problem-solving and ownership skills, with the ability to quickly understand unfamiliar systems and take initiative on new challenges.

• Familiarity with TypeScript is a plus, particularly for reading existing codebases and contributing directly to test and automation code.

• Additional valuable experience includes LiveKit or WebRTC, STT/TTS technologies such as Azure Speech, Google Cloud Speech, Deepgram, or Whisper, and AI/ML infrastructure such as vLLM, TGI, Triton, or Azure OpenAI.

• Experience with golden datasets, human-in-the-loop evaluation, Braintrust, Promptfoo, Cypress, Postman, AWS/Azure/GCP, Terraform, Pulumi, CloudFormation, Docker, or Kubernetes is also advantageous.

• Familiarity with product analytics, multilingual or localized testing, production monitoring, and error-tracking tools is a further plus.

Benefits

• Fully remote working environment with the opportunity to work from India within a distributed APAC-focused team.

• Full-time position within an engineering organization working on AI-powered products.

• High level of ownership and autonomy over QA strategy, automation, evaluation, and release quality.

• Opportunity to work hands-on with advanced AI technologies, including LLMs, RAG, conversational agents, and voice pipelines.

• Direct impact on the quality, reliability, and customer readiness of AI products.

• Opportunity to build and evolve testing frameworks, evaluation systems, and CI/CD quality infrastructure from the ground up.

• Fast-paced environment offering exposure to emerging AI technologies and challenging quality problems.

• Collaborative work across Engineering, Product, and QA functions.

• Professional growth through ownership of complex, evolving technical challenges.