[Remote] Staff AI Product Analyst

Note: The job is a remote job and is open to candidates in USA. Angi is a global company that has been transforming the home services industry for over 30 years. The Staff AI Product Analyst will be responsible for defining AI-native product analytics, creating evaluation frameworks, and ensuring alignment between human teams and AI agents in the product analytics function.ResponsibilitiesFlagship Journey Measurement: Partner with AI product, engineering, and domain experts to define north-star and guardrail metrics for flagship AI journeys, focusing on adoption, task completion, quality, and downstream business outcomesCausal Framing & Experimentation: Align closely with data science partners on test design, interpretation, and causal framing when new AI products shipData Layer Systems Thinking: Map how agents interact with metrics, raw tables, documentation, and APIs to ensure requirements for data and analytics engineering are explicit, structured, and testableEval Framework Architecture: Design and maintain formal evaluation sets, "golden datasets," rubrics, and regression checks for agent outputs to guarantee accuracy, completeness, policy safety, and usefulnessTaxonomy & Failure Mode Analysis: Systematically categorize wrong answers, tool misuse, context gaps, and data edge cases, feeding those insights back into prompts, retrieval (RAG), tools, and data modeling pipelinesPerformance Monitoring: Monitor live usage of Agentic AI analytics workflows—tracking success, abandonment, fallback patterns, and human override rates to maintain a prioritized engineering improvement backlogPlaybooks & Narrative: Own internal documentation defining when to use agents, required human checkpoints, and how to communicate uncertainty. Support enablement and leadership storytelling around agentic AI analyticsStrategic Collaboration: Partner with business leaders, the Principal Analytics Engineer, and data engineering teams to translate complex AI capabilities into measurable business outcomesDefining Success: Deliver recurring measurement artifacts (dashboards, reviews, or written readouts) that leadership uses to prioritize roadmaps, while continuously reducing high-severity incidents where agents produce incorrect metricsSkills10+ years of experience across product analytics, data science, or technical product management within a complex, high-scale operating environment1+ years of direct, hands-on experience working with AI/LLM systems in an applied context, with a specific focus on evaluating, trusting, and deploying them responsiblyDeep technical proficiency with Python, ML frameworks, data visualization/analytics platforms, and standard data management/governance practicesProven track record designing AI evaluation frameworks, including formal evals, golden datasets, production monitoring, or equivalent rigor in a related data disciplineExceptional communication skills with a demonstrated ability to translate highly technical AI insights and tradeoffs into clear narratives for non-technical business stakeholdersExperience building or evaluating agentic analytics workflows, orchestration frameworks, and familiarity with multi-agent systemsDemonstrated ability to operate fluidly in ambiguity, run structured experiments or pilots, and default to fast action and iterationA systems-oriented mindset with a healthy skepticism of AI hype, focusing strictly on organizational outcomes and driving technical investments that yield true business valueBenefitsThis position will be eligible for a competitive year end performance bonus & equity package.Full medical, dental, vision package to fit your needsFlexible vacation policy; work hard and take time when you need itPet discount plans & retirement plan with company match (401K)Company OverviewAngi is a technology-based tool that helps homeowners find home improvement and property renovation professionals. It was founded in 1995, and is headquartered in Indianapolis, Indiana, USA, with a workforce of 1001-5000 employees. Its website is https://www.angi.com/.Company H1B SponsorshipAngi has a track record of offering H1B sponsorships, with 2 in 2026, 17 in 2025, 39 in 2024, 15 in 2023, 32 in 2022. Please note that this does not guarantee sponsorship for this specific role.