[Remote] Senior AI QA Engineer with Python (Automation & Manual)

Note: The job is a remote job and is open to candidates in USA. EPAM Systems is a leading company in the tech industry, and they are seeking a skilled Senior AI QA Engineer with strong experience in both manual and automated testing. The ideal candidate will test a variety of AI-based applications, ensuring reliability and accuracy while contributing to the development of automation capabilities.ResponsibilitiesResearch and evolve automation frameworks in line with Gen AI tooling and best practicesDesign and automate the evaluation of Gen AI features — grounding, answer accuracy, determinism/reproducibility, precision, recall, and criteria recallBuild automated LLM test harnesses that scale evaluation beyond human-in-the-loopSelect and apply Gen AI evaluation frameworks, measuring answer quality and pipeline efficiencyPerform manual testing as needed to validate new features, integrations, and user storiesBuild and maintain test cases from requirements and user storiesTest applications that may include AI agents, APIs, databases, and other integrationsCollaborate with product, engineering, and operations teams to understand requirements and deployment environmentsTrack and report test results, defects, and quality metricsAssist with troubleshooting production issues; escalate risks as neededGuide and support team members, including onshore and offshore consultantsSkills3+ years of experience in software QA, with at least 1 year focused on testing AI agents, agentic solutions, or LLM-based systemsHands-on experience with both manual and automated testing of AI agents, including prompt/instruction testing and evaluation of agentic workflowsStrong programming skills in Python for test automation — pytest or equivalent, scripting, and AI/ML library integrationExperience with AI agent frameworks, prompt engineering, and evaluation metrics for LLM-based systemsDemonstrated experience in testing and evaluating Gen AI / LLM applications — grounding, answer accuracy, and hallucination/determinism checksApplied knowledge of Gen AI / LLM evaluation frameworks and metrics — precision, recall, criteria recall, and efficiencyExperience with issue and test management tools (e.g., Jira, QMetry, TestRail)Experience with version control systems and integrating tests into CI/CD pipelinesExperience using AI-powered tools for QA (e.g., GitHub Copilot, LLM-based test generation)Understanding of cloud environments, particularly AWSExcellent communication, collaboration, and leadership skillsStrong English communication skills (B2 level or higher)Experience with agentic AI platforms (e.g., LangChain, OpenAI Function Calling, or similar)Skills in AI safety, bias, and reliability testingBackground in test data generation for AI/ML systemsBenefitsInternational projects with top brandsWork with global teams of highly skilled, diverse peersHealthcare benefitsEmployee financial programsPaid time off and sick leaveUpskilling, reskilling and certification coursesUnlimited access to the LinkedIn Learning library and 22,000+ coursesGlobal career opportunitiesVolunteer and community involvement opportunitiesEPAM Employee GroupsAward-winning culture recognized by Glassdoor, Newsweek and LinkedInCompany OverviewEPAM leverages its core engineering expertise as a leading global product development and digital platform engineering services company. It was founded in 1993, and is headquartered in Newtown, Pennsylvania, USA, with a workforce of 10001+ employees. Its website is https://www.epam.com.Company H1B SponsorshipEPAM Systems has a track record of offering H1B sponsorships, with 11 in 2026, 120 in 2025, 172 in 2024, 232 in 2023, 373 in 2022, 359 in 2021, 502 in 2020. Please note that this does not guarantee sponsorship for this specific role.