[Remote] AI Validation Engineer

Note: The job is a remote job and is open to candidates in USA. Agility Partners is hiring for QA / AI Quality Engineer roles to support critical AI initiatives across its digital ecosystem. The role involves designing and implementing validation frameworks for AI-powered systems, ensuring outputs are accurate and reliable, and collaborating with engineering and product teams to improve system quality.ResponsibilitiesDesign and implement validation and QA frameworks for AI-powered systems, including analytics platforms, LLM-driven applications, and agentic workflowsEvaluate AI outputs for accuracy, grounding, consistency, and alignment with underlying data and expected behaviorTest and validate key components such as:RAG pipelines and retrieval qualityData pipelines and transformationsPrompt behavior and response consistencyAPI integrations and downstream system interactionsBuild automated testing frameworks for structured and unstructured AI systemsDefine and track metrics around accuracy, data integrity, performance, and system reliabilityIdentify discrepancies between AI outputs and source data or expected outcomesPartner closely with engineering, data, and product teams to improve system quality and reduce defectsSupport rapid iteration cycles by developing scalable evaluation and regression testing approachesMonitor AI systems post-deployment to detect drift, degradation, and performance issuesHelp implement guardrails to improve safety, reduce hallucinations, and ensure trustworthy outputsSupport auditability and traceability of AI-driven systems and insightsSkills3–7+ years of experience in QA, Software Testing, Data Validation, or AI/ML-focused quality engineeringStrong Python skills for automation, testing, and data validationExperience building automated test frameworks and validation pipelinesExperience testing AI/ML or data-driven systems (LLMs, analytics platforms, or similar)Strong SQL or data querying skills and ability to validate large datasetsUnderstanding of data pipelines, transformations, and data quality validationExperience validating APIs and service integrationsAbility to analyze logs, outputs, and system behavior to diagnose issuesComfortable working in fast-paced, agile environments with evolving requirementsExperience with LLM evaluation, prompt testing, or RAG-based systemsFamiliarity with agentic workflows or orchestration patternsExposure to LLMOps, AI evaluation tooling, or observability frameworksExperience with model monitoring, drift detection, or performance trackingFamiliarity with analytics platforms or BI toolsUnderstanding of explainability techniques, governance, or risk in AI systemsAwareness of cost, latency, and performance considerations in AI solutionsCompany OverviewAgility Partners is an IT recruiting firm that provides consulting services for businesses. It was founded in 2017, and is headquartered in Columbus, Ohio, USA, with a workforce of 51-200 employees. Its website is https://agilitypartners.io.Company H1B SponsorshipAgility Partners has a track record of offering H1B sponsorships, with 3 in 2025, 1 in 2024. Please note that this does not guarantee sponsorship for this specific role.