Principal AI Governance Scientist - Artificial Intelligence
Position Purpose: Leads the technical evaluation and assurance efforts within our AI Governance team. Establishes enterprise-grade, decision-relevant methodologies for red teaming, testing, and evaluating AI systems across traditional ML, Generative AI, and Agentic AI applications, ensuring evaluations directly inform AI governance decisions, deployment readiness, and ongoing oversight.
ā¢ Develops reproducible frameworks to measure AI value, user impact, and broader outcomes to support responsible scaling, risk acceptance, and investment decisions
ā¢ Designs rigorous evaluation methodologies for assessing AI system performance, safety, reliability, and alignment with intended use across the AI lifecycle, from development through deployment and monitoring
ā¢ Develops criteria and benchmarks to determine whether existing evaluations are adequate and sufficient for different AI applications and risk profiles
ā¢ Designs and executes comprehensive red team exercises to identify vulnerabilities, failure modes, and unintended behaviors across diverse AI systems and devise solutions to address them
ā¢ Develops rigorous evaluation methodologies and criteria to assess whether existing evaluations are adequate and sufficient for different AI applications and risk profiles
ā¢ Establishes standards for evaluation coverage, rigor, and documentation across the AI lifecycle
ā¢ Establishes reproducible methodologies for measuring business value, user impact, and societal outcomes of AI systems using causal inference and experimental design
ā¢ Advances the scientific understanding of AI evaluation and safety through white papers and trainings
ā¢ Provides technical leadership and mentorship to scientists, engineers, and compliance professionals while building organizational evaluation capabilities
ā¢ Stays at the forefront of AI safety research and identify novel risks emerging from advanced AI capabilities, particularly in frontier models
ā¢ Translates complex technical findings into actionable recommendations for leadership, governance boards, and cross-functional teams
ā¢ Collaborates with external researchers, institutions, and industry partners to advance evaluation methodology and contribute to the broader AI safety community
ā¢ Performs other duties as assigned
ā¢ Complies with all policies and standards
Education/Experience: Bachelor's Degree Computer Science, Machine Learning, Statistics, or related quantitative field; or equivalent experience required. Master's Degree preferred
Technical Skills:
ā¢ 6+ years AI/ML research, including 3+ years focused on model evaluation, safety, or robustness required. 8+ years preferred
ā¢ Deep technical expertise in modern AI systems with hands-on experience evaluating large language models, generative AI, and/or agentic systems required
ā¢ Proven track record designing rigorous evaluation methodologies and publication record required
ā¢ Strong foundation in statistical methods, experimental design, causal inference, and excellent Python programming skills with ML frameworks required
ā¢ Familiarity with Python-based AI/ML stack using PyTorch and Databricks, with agentic AI frameworks (LangChain, LlamaIndex, LangGraph, AutoGen, CrewAI) for single- and multi-agent systems. Strong focus on LLM observability, MLOps, and evaluation using LangSmith, MLflow, Weights & Biases, Datadog, OpenTelemetry, and testing frameworks like DeepEval and LangTest
Pay Range: $134,600.00 - $249,000.00 per year
Centene offers a comprehensive benefits package including: competitive pay, health insurance, 401K and stock purchase plans, tuition reimbursement, paid time off plus holidays, and a flexible approach to work with remote, hybrid, field or office work schedules. Actual pay will be adjusted based on an individual's skills, experience, education, and other job-related factors permitted by law, including full-time or part-time status. Total compensation may also include additional forms of incentives. Benefits may be subject to program eligibility.
Apply tot his job
Apply To this Job
ā¢ Develops reproducible frameworks to measure AI value, user impact, and broader outcomes to support responsible scaling, risk acceptance, and investment decisions
ā¢ Designs rigorous evaluation methodologies for assessing AI system performance, safety, reliability, and alignment with intended use across the AI lifecycle, from development through deployment and monitoring
ā¢ Develops criteria and benchmarks to determine whether existing evaluations are adequate and sufficient for different AI applications and risk profiles
ā¢ Designs and executes comprehensive red team exercises to identify vulnerabilities, failure modes, and unintended behaviors across diverse AI systems and devise solutions to address them
ā¢ Develops rigorous evaluation methodologies and criteria to assess whether existing evaluations are adequate and sufficient for different AI applications and risk profiles
ā¢ Establishes standards for evaluation coverage, rigor, and documentation across the AI lifecycle
ā¢ Establishes reproducible methodologies for measuring business value, user impact, and societal outcomes of AI systems using causal inference and experimental design
ā¢ Advances the scientific understanding of AI evaluation and safety through white papers and trainings
ā¢ Provides technical leadership and mentorship to scientists, engineers, and compliance professionals while building organizational evaluation capabilities
ā¢ Stays at the forefront of AI safety research and identify novel risks emerging from advanced AI capabilities, particularly in frontier models
ā¢ Translates complex technical findings into actionable recommendations for leadership, governance boards, and cross-functional teams
ā¢ Collaborates with external researchers, institutions, and industry partners to advance evaluation methodology and contribute to the broader AI safety community
ā¢ Performs other duties as assigned
ā¢ Complies with all policies and standards
Education/Experience: Bachelor's Degree Computer Science, Machine Learning, Statistics, or related quantitative field; or equivalent experience required. Master's Degree preferred
Technical Skills:
ā¢ 6+ years AI/ML research, including 3+ years focused on model evaluation, safety, or robustness required. 8+ years preferred
ā¢ Deep technical expertise in modern AI systems with hands-on experience evaluating large language models, generative AI, and/or agentic systems required
ā¢ Proven track record designing rigorous evaluation methodologies and publication record required
ā¢ Strong foundation in statistical methods, experimental design, causal inference, and excellent Python programming skills with ML frameworks required
ā¢ Familiarity with Python-based AI/ML stack using PyTorch and Databricks, with agentic AI frameworks (LangChain, LlamaIndex, LangGraph, AutoGen, CrewAI) for single- and multi-agent systems. Strong focus on LLM observability, MLOps, and evaluation using LangSmith, MLflow, Weights & Biases, Datadog, OpenTelemetry, and testing frameworks like DeepEval and LangTest
Pay Range: $134,600.00 - $249,000.00 per year
Centene offers a comprehensive benefits package including: competitive pay, health insurance, 401K and stock purchase plans, tuition reimbursement, paid time off plus holidays, and a flexible approach to work with remote, hybrid, field or office work schedules. Actual pay will be adjusted based on an individual's skills, experience, education, and other job-related factors permitted by law, including full-time or part-time status. Total compensation may also include additional forms of incentives. Benefits may be subject to program eligibility.
Apply tot his job
Apply To this Job