Quality Assurance (QA) - Lead - MiDAS

Your tasks

Roles & Responsibilities : AI Quality Strategy: Develop and own the evaluation framework for GenAI solutions, focusing on** Faithfulness, Relevancy, and Hallucination detection** using LLM-as-a-judge frameworks.

• Hybrid Test Automation: Architect a dual-layered automation suite:

Deterministic: E2E UI (Playwright) and API testing (Pytest/Requests).

• Probabilistic: Automated evaluation of non-deterministic LLM outputs.

• Shift-Left Integration: Embed automated quality checks directly into** GitHub Workflows**, enabling seamless CI/CD.

Performance & Resilience: Lead JMeter-based performance testing.

Your profile

Educational qualification:

• Experience: 8+ years in Software QA
• Problem Solving: Ability to define "quality" in an ambiguous, non-deterministic AI landscape.
• Education: Bachelor’s or Master’s degree in Computer Science, Software Engineering, or a related field.

Experience :

• 8+ years in Software QA

Mandatory/requires Skills : Automation & Tooling

• Python Mastery: Expert-level Python skills for building custom test tooling and automation scripts.
• Testing Stack: Hands-on proficiency with** Pytest**(API),** Playwright**(E2E), and** JMeter** (Performance).
• DevOps: Advanced experience designing and maintaining** GitHub Actions/Workflows** for automated test execution.

Core AI & LLM Expertise

• Learning Agility in GenAI: High capability and interest in rapidly mastering AI evaluation concepts. You should be prepared to quickly upskill in automated metrics for LLMs (such as Faithfulness, Relevancy, and Groundedness).
• Exposure to LLM Logic: Basic familiarity with how LLMs function (e.g., prompting, context windows). You should be comfortable exploring and implementing "LLM-as-a-Judge" strategies, where high-reasoning models help grade application-specific outputs.
• Orientation toward RAG Systems: Interest in understanding the mechanics of Retrieval-Augmented Generation (RAG). You will be responsible for defining how we validate the accuracy of data retrieved from our engineering context catalogues and vector databases.
• Data-Driven Quality Mindset: A strong desire to move beyond binary "Pass/Fail" results toward probabilistic quality monitoring, utilizing tools like Langfuse to analyze live traces and performance trends.

Preferred Skills :