[Remote] AI Test Engineer II

Note: The job is a remote job and is open to candidates in USA. TREND Health Partners is a tech-enabled payment integrity company focused on improving healthcare access through collaboration between payers and providers. The AI Test Engineer II is responsible for validating the quality and reliability of AI powered systems, including designing test strategies and evaluating model performance across the platform.ResponsibilitiesDesign and execute test strategies for AI/ML models, including functional, regression, and scenario-based testingValidate accuracy, consistency, and reliability of AI-generated outputs (e.g., summaries, classifications, recommendations)Develop test cases for edge cases, model drift, and unexpected behavior scenariosEvaluate model performance against defined success criteria and business requirementsTest agent-driven workflows (multi-step AI processes, automation pipelines, orchestration logic)Validate interactions between AI agents, APIs, and downstream systemsIdentify failure points in end-to-end AI workflows and recommend improvementsValidate training, validation, and test datasets for completeness, accuracy, and bias risksPerform data validation testing across pipelines to ensure integrity of inputs and outputsPartner with Data Engineering to detect anomalies, inconsistencies, and gapsBuild and maintain automated test frameworks for AI-enabled systemsDevelop scripts and tools to support repeatable testing of AI outputs at scaleIntegrate testing into CI/CD pipelines to enable continuous validationHelp define and validate guardrails for AI systems (accuracy thresholds, explainability, fallback logic)Identify risks related to hallucinations, bias, and model driftPartner with Engineering and AI teams to improve reliability and safetyWork cross-functionally with Engineering, QA, AI, Product, and Operations teamsParticipate in sprint planning, backlog refinement, and release validationContribute to testing standards, best practices, and documentationSupport root cause analysis for AI-related defects and production issuesSkillsBachelor's degree in Computer Science, Data Science, Engineering, or related field (or equivalent experience)Experience in QA, testing, or a related engineering roleExperience testing APIs, data pipelines, or distributed systemsStrong understanding of software testing principles and test automationExperience working in Agile development environmentFamiliarity with AI/ML concepts (model evaluation, training vs inference, prompt behavior, etc.)Experience with test automation toolsStrong SQL/data validation skillsExperience working with cloud-based systems (AWS or Databricks preferred)Ability to analyze system behavior and identify root causesExperience testing AI/ML models or data-driven systemsExperience validating AI-generated outputs or prompt-based systemsFamiliarity with LLMs, agent workflows, or AI-assisted coding toolsHealthcare or regulated data environment experience (HIPAA, compliance)Experience working with CI/CD and DevOps pipelinesBenefitsCompetitive salariesHighly valued health insuranceA 401(k) plan with employer matchPaid parental leaveCompany OverviewTrend Health Partners is an employee owned company that provides payment accuracy services for people. It was founded in 2018, and is headquartered in Hunt Valley, Maryland, USA, with a workforce of 201-500 employees. Its website is https://www.trendhealthpartners.com.Company H1B SponsorshipTrend Health Partners has a track record of offering H1B sponsorships, with 1 in 2025. Please note that this does not guarantee sponsorship for this specific role.