Tech Lead Data Scientist, AI Evaluation – Monitoring
Job Description:
• Principal technical expert for how Geisinger evaluates, monitors, and optimizes AI systems
• Set technical direction for AI evaluation across a growing portfolio
• Provide technical leadership to a team of data analysts
• Partner with AI program teams to improve validation and monitoring methods
• Design validation studies and monitoring plans for AI systems
• Instrument production monitoring and define measurable metrics
• Mentor analysts on methodology and technical rigor
Requirements:
• 6+ years in data science, statistics, ML engineering, or applied quantitative research
• Strong foundation in experimental design and causal inference
• Hands-on experience designing and running model evaluation studies in real production settings
• Experience evaluating LLM or generative AI systems
• Proven ability to translate ambiguous failure modes into concrete evaluation designs
• Strong fluency in Python and SQL
• Experience with fairness and equity evaluation for ML systems
• MS or PhD in a quantitative field preferred
Benefits:
• Healthcare benefits from day one
• Vision insurance
• Dental insurance
• Domestic partner benefits
• Collaboration and cooperation atmosphere
Apply tot his job
Apply To this Job
• Principal technical expert for how Geisinger evaluates, monitors, and optimizes AI systems
• Set technical direction for AI evaluation across a growing portfolio
• Provide technical leadership to a team of data analysts
• Partner with AI program teams to improve validation and monitoring methods
• Design validation studies and monitoring plans for AI systems
• Instrument production monitoring and define measurable metrics
• Mentor analysts on methodology and technical rigor
Requirements:
• 6+ years in data science, statistics, ML engineering, or applied quantitative research
• Strong foundation in experimental design and causal inference
• Hands-on experience designing and running model evaluation studies in real production settings
• Experience evaluating LLM or generative AI systems
• Proven ability to translate ambiguous failure modes into concrete evaluation designs
• Strong fluency in Python and SQL
• Experience with fairness and equity evaluation for ML systems
• MS or PhD in a quantitative field preferred
Benefits:
• Healthcare benefits from day one
• Vision insurance
• Dental insurance
• Domestic partner benefits
• Collaboration and cooperation atmosphere
Apply tot his job
Apply To this Job