[Remote] Senior Quality Assurance Engineer

Note: The job is a remote job and is open to candidates in USA. Peach Pilot is a company that transforms how businesses run by building a platform that connects operations through AI. The Senior Quality Assurance Engineer will be responsible for establishing testing frameworks and ensuring the quality of AI-generated outputs before they reach clients, playing a crucial role in maintaining trust and reliability in the company's deliverables.ResponsibilitiesEstablish the testing framework: unit, integration, end-to-end, and AI-specific evaluation pipelines using Playwright and VitestDefine quality standards, test coverage requirements, and documentation practices in partnership with the Lead EngineerAudit the existing platform and identify the highest-risk surfaces before the next client deploymentDesign evaluation frameworks for non-deterministic LLM outputs — including prompt regression testing, model drift detection, and output quality scoringBuild automated test suites for the agent orchestration layer, including governance-agent audit-trail integrity and human-override behaviorValidate the Company Brain (Memgraph + Qdrant) for data accuracy, retrieval quality, and failure modes under real enterprise data including entity resolution across systems and temporal data patternsTest the Analysis Engine pipeline that surfaces Company X-Ray findings ensuring insights are not just technically accurate but reliable enough to present to a clientOwn end-to-end testing of the data ingestion pipelines that connect to client systems CRM, email, calls, calendars, documents, financial systems through Nango's 700+ connector integration layerTest multi-model routing logic to confirm cost-optimized task allocation behaves correctly across LLM providers via LiteLLMValidate streaming response handling, latency thresholds, and graceful degradation when a model is unavailable or slowOwn file ingestion pipeline testing (Word, Excel, PowerPoint, PDF) including encryption, formatting edge cases, and audit-trail continuitySkills7+ years of QA engineering experience, with at least 3 years in a senior or lead capacity where you shaped process and standards not just executed themYou have tested AI/LLM-powered applications. You understand prompt sensitivity, output variance, and how to build eval pipelines that catch regressions across model updatesYou speak in ownership: you've built the eval pipeline, owned model quality, gated the release — not just run someone else's test suiteYou write test code. Python is your primary tool. You have built and maintained CI/CD-integrated test suites, and you don't wait for someone to file a bug to find oneHands-on experience with Playwright and Vitest in a production environment and you've built automation frameworks from scratch, not just inherited themComfortable testing complex API chains, async/streaming responses, and multi-service workflows. Data pipelines and knowledge graph outputs don't intimidate youYou test for confusion and trust failure not just broken functionality. Your end users are non-technical executives, and you advocate for themUS-based, able to overlap roughly 5 hours per day with EDT, and available for full-time contract hoursYou have experience with LLM evaluation frameworks (e.g., LangSmith, DeepEval, Promptfoo, RAGAS, or custom eval pipelines)You have tested agent frameworks or orchestration layers in a production environmentYou have a background in a regulated industry (insurance, finance, healthcare) where audit-trail integrity is non-negotiableYou have worked alongside Forward Deployed or solutions engineering teams and understand field deployment riskBenefitsContract-to-hire possible if the engagement goes wellCompany OverviewMost AI tools are built for engineers. Peach Pilot is built for everyone else. We're an enterprise AI company based in Atlanta, GA. It was founded in 2026, and is headquartered in Atlanta, Georgia, US, with a workforce of 2-10 employees. Its website is https://www.peachpilot.ai.