[Remote] Senior Manager, AI Performance & Operations

Note: The job is a remote job and is open to candidates in USA. Devoted Health is on a mission to improve the health and well-being of older Americans by providing integrated healthcare solutions. The Senior Manager, AI Performance & Operations will oversee the performance and continuous improvement of AI workflows, ensuring they meet operational outcomes and quality standards.ResponsibilitiesPost-Launch Accountability: Own the health of major voice and text AI workflows, ensuring business, quality, safety, and operational outcomes are metPortfolio Oversight: Maintain a current view of all major workflows, including status, maturity, failure modes, and next actionsPerformance & Dashboards: Monitor weekly indicators (containment, escalation, complaints) and oversee production dashboards to track behavior independentlyDiagnosis & Incident Response: Lead response for AI regressions; distinguish between model, prompt, or orchestration issues and drive post-mortems to closureContinuous Improvement & Optimization: Run structured cycles (hypothesis, testing, rollout/rollback) for "mini-launches" like model swaps, prompt changes, and tool updates. Recommend evidence-based accelerations, such as model migrations or deeper tool-orchestration changesScalable Evaluation: Design eval systems using golden datasets, LLM-as-judge workflows, and human review queues for production and pre-productionExpert Review: Guide new workflows during incubation to ensure they launch with proper instrumentation, eval coverage, and safety modelsExecutive Communication: Translate technical changes into plain language for leadership, explaining what broke, why it matters, and who is fixing itSkillsStrong production experience with LLMs, RAG, voice AI, agentic orchestration, or equivalent ML systemsDemonstrated experience designing or operating rubrics, scoring frameworks, regression tests, and production-sampling workflowsA clear mental model for distinguishing failure modes, such as model reasoning, tool failure, retrieval issues, and workflow logicComfort with active experimentation—including prompt/context tuning, retrieval redesign, and safe model migrations—beyond simple issue triageExcellent cross-functional judgment and the ability to communicate clearly in ambiguous, high-accountability environmentsDeep personal accountability with a drive to own results rather than just producing recommendationsExperience in healthcare, regulated workflows, contact center operations, or high-stakes service quality/reliability functionsBenefitsEmployer sponsored health, dental and vision plan with low or no premiumGenerous paid time off$100 monthly mobile or internet stipendStock options for all employeesBonus eligibility for all roles excluding Director and above; Commission eligibility for Sales rolesParental leave program401K programOur total rewards package is for full time employees only. Intern and Contract positions are not eligible.Company OverviewDevoted Health is a healthcare company serving seniors and giving them a health care plan with personal guides and world-class technology. It was founded in 2017, and is headquartered in Waltham, Massachusetts, USA, with a workforce of 1001-5000 employees. Its website is https://devoted.com.Company H1B SponsorshipDevoted Health has a track record of offering H1B sponsorships, with 1 in 2021. Please note that this does not guarantee sponsorship for this specific role.