[Remote] QA Engineer, AI Products
Note: The job is a remote job and is open to candidates in USA. MDCalc is a leading medical reference tool used by clinicians worldwide, and they are seeking a QA Engineer to enhance their AI product team. This role focuses on ensuring the quality and reliability of AI-powered features, particularly in testing LLM-based systems, while collaborating with cross-functional teams to define quality metrics and testing strategies.
Responsibilities
ā¢ Design and execute test strategies for LLM-powered features, including prompt regression testing, output evaluation, and hallucination detection
ā¢ Build and maintain automated evaluation pipelines (eval sets, golden datasets, LLM-as-judge frameworks) to catch quality regressions in non-deterministic outputs
ā¢ Perform black-box and exploratory testing of MDCalc's AI features across web and mobile, with particular attention to clinical accuracy, safety, and edge cases
ā¢ Define quality metrics for AI outputs (accuracy, faithfulness, relevance, safety, latency, cost) and establish thresholds for release readiness
ā¢ Collaborate cross-functionally with engineers, product managers, ML/AI engineers, and clinical reviewers to define what 'good' looks like for AI responses
ā¢ Investigate and triage AI failure modes, distinguishing model issues, prompt issues, retrieval issues, and integration bugs
ā¢ Participate in team discussions, offering feedback on testability, risks, prompt design, and guardrails
ā¢ Help develop QA strategies to expand future testing capacity, automation, and evaluation coverage as the AI product surface grows
Skills
ā¢ 5+ years of experience in software QA, with at least 1 year of hands-on testing of LLM-based or AI/ML-powered features
ā¢ Strong understanding of QA principles, test case creation/documentation, and best practices for both deterministic and non-deterministic systems
ā¢ Hands-on experience with LLM tooling and concepts: prompt engineering, RAG systems, evaluation frameworks (e.g., Promptfoo, Braintrust, LangSmith, DeepEval, Ragas, OpenAI Evals), and LLM APIs (OpenAI, Anthropic, etc.)
ā¢ Experience designing automated qualitative evaluation approaches, including LLM-as-judge, rubric-based scoring, semantic similarity checks, and golden dataset regression testing
ā¢ Proficiency with test automation tools, with a focus on Playwright
ā¢ Strong SQL skills for data validation, test data creation, and verifying data integrity across systems
ā¢ Familiarity with token usage, latency profiling, and cost monitoring as quality signals
ā¢ Eagerness to learn quickly and a positive, solutions-oriented attitude
ā¢ Clear and concise communicator, able to surface issues, blockers, and risks effectively when communicating ambiguous or probabilistic failures
ā¢ Self-motivated, proactive, and able to manage time and priorities independently
Benefits
ā¢ Medical, Dental, & Vision Coverage, with option to extend to your dependents
ā¢ Company-sponsored short-term insurance
ā¢ Fully-paid 8 week parental leave, after 6 months of employment
ā¢ Company-sponsored 401k, after 3 months of employment
ā¢ Unlimited vacation for salaried roles - we trust you to take the time you need
ā¢ Bi-annual company offsites to connect, reflect, and plan together
ā¢ Work from home monthly stipend
ā¢ A culture of fun and motivated team members who believe in a greater mission here at MDCalc
Company Overview
ā¢ MDCalc is used by over 2/3 of US physicians, provides free and access to 800+ medical scores, calculations and algorithms. It was founded in 2005, and is headquartered in New York, New York, USA, with a workforce of 11-50 employees. Its website is https://www.mdcalc.com.
Apply To This Job
Responsibilities
ā¢ Design and execute test strategies for LLM-powered features, including prompt regression testing, output evaluation, and hallucination detection
ā¢ Build and maintain automated evaluation pipelines (eval sets, golden datasets, LLM-as-judge frameworks) to catch quality regressions in non-deterministic outputs
ā¢ Perform black-box and exploratory testing of MDCalc's AI features across web and mobile, with particular attention to clinical accuracy, safety, and edge cases
ā¢ Define quality metrics for AI outputs (accuracy, faithfulness, relevance, safety, latency, cost) and establish thresholds for release readiness
ā¢ Collaborate cross-functionally with engineers, product managers, ML/AI engineers, and clinical reviewers to define what 'good' looks like for AI responses
ā¢ Investigate and triage AI failure modes, distinguishing model issues, prompt issues, retrieval issues, and integration bugs
ā¢ Participate in team discussions, offering feedback on testability, risks, prompt design, and guardrails
ā¢ Help develop QA strategies to expand future testing capacity, automation, and evaluation coverage as the AI product surface grows
Skills
ā¢ 5+ years of experience in software QA, with at least 1 year of hands-on testing of LLM-based or AI/ML-powered features
ā¢ Strong understanding of QA principles, test case creation/documentation, and best practices for both deterministic and non-deterministic systems
ā¢ Hands-on experience with LLM tooling and concepts: prompt engineering, RAG systems, evaluation frameworks (e.g., Promptfoo, Braintrust, LangSmith, DeepEval, Ragas, OpenAI Evals), and LLM APIs (OpenAI, Anthropic, etc.)
ā¢ Experience designing automated qualitative evaluation approaches, including LLM-as-judge, rubric-based scoring, semantic similarity checks, and golden dataset regression testing
ā¢ Proficiency with test automation tools, with a focus on Playwright
ā¢ Strong SQL skills for data validation, test data creation, and verifying data integrity across systems
ā¢ Familiarity with token usage, latency profiling, and cost monitoring as quality signals
ā¢ Eagerness to learn quickly and a positive, solutions-oriented attitude
ā¢ Clear and concise communicator, able to surface issues, blockers, and risks effectively when communicating ambiguous or probabilistic failures
ā¢ Self-motivated, proactive, and able to manage time and priorities independently
Benefits
ā¢ Medical, Dental, & Vision Coverage, with option to extend to your dependents
ā¢ Company-sponsored short-term insurance
ā¢ Fully-paid 8 week parental leave, after 6 months of employment
ā¢ Company-sponsored 401k, after 3 months of employment
ā¢ Unlimited vacation for salaried roles - we trust you to take the time you need
ā¢ Bi-annual company offsites to connect, reflect, and plan together
ā¢ Work from home monthly stipend
ā¢ A culture of fun and motivated team members who believe in a greater mission here at MDCalc
Company Overview
ā¢ MDCalc is used by over 2/3 of US physicians, provides free and access to 800+ medical scores, calculations and algorithms. It was founded in 2005, and is headquartered in New York, New York, USA, with a workforce of 11-50 employees. Its website is https://www.mdcalc.com.
Apply To This Job