AI Product Analyst - AI Receptionist Team
At MEDvidi, we are scaling AI-powered systems across critical parts of our product and operations.
As these systems become more autonomous and deeply integrated into user workflows, understanding how they behave in production becomes a core business challenge.
We are hiring a Product Analyst to help us evaluate, monitor, and improve AI agents and LLM-powered workflows operating at scale.
This role is focused on analyzing real AI system behavior in production environments:
understanding product and operational impact,
identifying failure patterns on generational tasks,
measuring quality and reliability of outputs,
helping teams make accurate decisions under ambiguity.
Responsibilities
Analyze production behavior of AI agents, automation systems, and LLM-powered workflows,
Investigate hallucinations, inconsistencies, failure modes, and quality degradation across automated user journeys,
Work directly with logs, conversations, raw system outputs, and operational datasets to identify hidden patterns and actionable insights,
Design evaluation approaches and analytical frameworks for measuring AI system quality and reliability,
Build monitoring systems and dashboards for AI performance visibility,
Identify operational and product risks introduced by automation,
Measure the impact of AI systems on business metrics, operational efficiency, and user experience,
Detect data quality issues, shifts, and anomalies affecting AI system performance,
Partner closely with Product, Engineering, QA, and Operations teams on AI-related initiatives,
Support rapid decision-making in fast-changing and ambiguous environments.
Requirements
3+ years of experience in analytical roles related to AI systems, ML products, automation platforms, or complex production systems,
Strong Python skills for deep analytical investigations and working with large-scale noisy text data,
Advanced SQL skills,
Understanding of LLM limitations,hallucinations, evaluation challenges, prompt sensitivity, model trade-offs, and AI failure modes,
Experience building monitoring systems, analytical frameworks, or operational dashboards for production systems,
Ability to work with incomplete, ambiguous, and rapidly changing data environments,
Strong ownership mindset and ability to independently drive investigations and analytical conclusions,
Ability to connect AI system quality with business and operational impact,
Fluent Russian; English level B2 or higher,
Ability to work CET-overlapping hours.
Nice to Have
Experience with conversational AI or AI agents,
Experience evaluating LLM outputs or annotation quality,
Experience with AI quality metrics and evaluation methodologies,
Experience working with Langchain infrastructure,
Experience with ML data pipelines or synthetic data generation,
Background in anomaly detection.
What we offer
Work on a modern AI product where quality is a strategic function, not a formality
Real influence on product direction and engineering practices
High-impact work in a product-driven medical tech company
Startup environment: fast iteration, high ownership, minimal bureaucracy
Competitive compensation and growth opportunities
Benefits
Health insurance after probation period
Sports & wellness compensation
Fully remote, long-term B2B collaboration
19 working days of vacation per year
3 additional wellness days per year
Paid sick leave for the first 5 working days
Gifts to employees for important events
Career Development
Personalized English lessons via Preply
Ready to make an impact? Send us your profile - let’s build something meaningful together.
Apply To This Job
As these systems become more autonomous and deeply integrated into user workflows, understanding how they behave in production becomes a core business challenge.
We are hiring a Product Analyst to help us evaluate, monitor, and improve AI agents and LLM-powered workflows operating at scale.
This role is focused on analyzing real AI system behavior in production environments:
understanding product and operational impact,
identifying failure patterns on generational tasks,
measuring quality and reliability of outputs,
helping teams make accurate decisions under ambiguity.
Responsibilities
Analyze production behavior of AI agents, automation systems, and LLM-powered workflows,
Investigate hallucinations, inconsistencies, failure modes, and quality degradation across automated user journeys,
Work directly with logs, conversations, raw system outputs, and operational datasets to identify hidden patterns and actionable insights,
Design evaluation approaches and analytical frameworks for measuring AI system quality and reliability,
Build monitoring systems and dashboards for AI performance visibility,
Identify operational and product risks introduced by automation,
Measure the impact of AI systems on business metrics, operational efficiency, and user experience,
Detect data quality issues, shifts, and anomalies affecting AI system performance,
Partner closely with Product, Engineering, QA, and Operations teams on AI-related initiatives,
Support rapid decision-making in fast-changing and ambiguous environments.
Requirements
3+ years of experience in analytical roles related to AI systems, ML products, automation platforms, or complex production systems,
Strong Python skills for deep analytical investigations and working with large-scale noisy text data,
Advanced SQL skills,
Understanding of LLM limitations,hallucinations, evaluation challenges, prompt sensitivity, model trade-offs, and AI failure modes,
Experience building monitoring systems, analytical frameworks, or operational dashboards for production systems,
Ability to work with incomplete, ambiguous, and rapidly changing data environments,
Strong ownership mindset and ability to independently drive investigations and analytical conclusions,
Ability to connect AI system quality with business and operational impact,
Fluent Russian; English level B2 or higher,
Ability to work CET-overlapping hours.
Nice to Have
Experience with conversational AI or AI agents,
Experience evaluating LLM outputs or annotation quality,
Experience with AI quality metrics and evaluation methodologies,
Experience working with Langchain infrastructure,
Experience with ML data pipelines or synthetic data generation,
Background in anomaly detection.
What we offer
Work on a modern AI product where quality is a strategic function, not a formality
Real influence on product direction and engineering practices
High-impact work in a product-driven medical tech company
Startup environment: fast iteration, high ownership, minimal bureaucracy
Competitive compensation and growth opportunities
Benefits
Health insurance after probation period
Sports & wellness compensation
Fully remote, long-term B2B collaboration
19 working days of vacation per year
3 additional wellness days per year
Paid sick leave for the first 5 working days
Gifts to employees for important events
Career Development
Personalized English lessons via Preply
Ready to make an impact? Send us your profile - let’s build something meaningful together.
Apply To This Job