Human Evaluation & Content Quality Vendor Operations Manager
Location: Mountain View, CA (Hybrid)
Duration: 5 months contract
Job Description:
ā¢ As a Human Evaluation & Content Quality Vendor Operations Manager, you will play a key role in scaling and optimizing our global human evaluation ecosystem - the backbone of GenAI product quality. You'll partner with cross-functional teams and vendor partners to enhance data quality, improve rater accuracy, and streamline evaluation workflows that shape how our AI models learn and perform.
ā¢ This role blends operational rigor, quality management, and strategic vendor partnership to deliver measurable improvements in human evaluation outcomes and operational efficiency.
Responsibilities:
1. Drive Quality Improvements
ā¢ Partner with global vendors to design and implement strategies that raise quality scores across human evaluation workflows.
ā¢ Conduct root-cause analysis to uncover quality drivers and pilot innovative methodologies that go beyond traditional training approaches.
ā¢ Hold vendors accountable for sustained performance improvement through structured governance and ongoing enablement.
2. Support Rating & Calibration Programs
ā¢ Collaborate with the Rater Operations and Human Evaluation teams to organize and maintain rating question taxonomies and calibration materials.
ā¢ Lead or support calibration discussions with vendors to ensure consistent interpretation of rating guidelines and alignment with evolving model evaluation criteria.
ā¢ Facilitate feedback loops that enhance evaluator understanding and maintain output consistency across vendor sites.
3. Optimize Capacity Management
ā¢ Monitor and analyze vendor performance metrics to proactively identify bottlenecks and inefficiencies.
ā¢ Work with vendors to optimize rater availability and scheduling, ensuring adequate coverage for business needs while maintaining throughput and quality.
ā¢ Partner with internal teams on forecasting, backlog reduction, and operational readiness initiatives.
4. Performance Management & Reporting
ā¢ Track and report on key vendor KPIs (quality, timeliness, utilization, throughput).
ā¢ Analyze performance data to identify trends, highlight improvement areas, and communicate insights to internal stakeholders.
ā¢ Contribute to quarterly business reviews and strategic planning with vendor partners.
Experience:
ā¢ 7+ years of experience in Vendor Operations, Business Operations, or Program Management, ideally within a tech, AI, or content evaluation environment.
ā¢ Proven success managing and improving performance across multiple vendor partners.
ā¢ Strong analytical and problem-solving skills, with experience using data to drive operational decisions.
ā¢ Excellent communication and negotiation skills, with the ability to influence external partners and cross-functional stakeholders.
ā¢ Highly organized with a proactive approach to issue identification and resolution.
Skills:
ā¢ Human Evaluation
ā¢ Vendor Operations
ā¢ Content
ā¢ Root-cause
ā¢ Rater Operations
Education:
ā¢ Bachelor's degree.
About US Tech Solutions:
US Tech Solutions is a global staff augmentation firm providing a wide range of talent on-demand and total workforce solutions. To know more about US Tech Solutions, please visit www.ustechsolutions.com.
US Tech Solutions is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.
Apply tot his job
Apply To this Job
Duration: 5 months contract
Job Description:
ā¢ As a Human Evaluation & Content Quality Vendor Operations Manager, you will play a key role in scaling and optimizing our global human evaluation ecosystem - the backbone of GenAI product quality. You'll partner with cross-functional teams and vendor partners to enhance data quality, improve rater accuracy, and streamline evaluation workflows that shape how our AI models learn and perform.
ā¢ This role blends operational rigor, quality management, and strategic vendor partnership to deliver measurable improvements in human evaluation outcomes and operational efficiency.
Responsibilities:
1. Drive Quality Improvements
ā¢ Partner with global vendors to design and implement strategies that raise quality scores across human evaluation workflows.
ā¢ Conduct root-cause analysis to uncover quality drivers and pilot innovative methodologies that go beyond traditional training approaches.
ā¢ Hold vendors accountable for sustained performance improvement through structured governance and ongoing enablement.
2. Support Rating & Calibration Programs
ā¢ Collaborate with the Rater Operations and Human Evaluation teams to organize and maintain rating question taxonomies and calibration materials.
ā¢ Lead or support calibration discussions with vendors to ensure consistent interpretation of rating guidelines and alignment with evolving model evaluation criteria.
ā¢ Facilitate feedback loops that enhance evaluator understanding and maintain output consistency across vendor sites.
3. Optimize Capacity Management
ā¢ Monitor and analyze vendor performance metrics to proactively identify bottlenecks and inefficiencies.
ā¢ Work with vendors to optimize rater availability and scheduling, ensuring adequate coverage for business needs while maintaining throughput and quality.
ā¢ Partner with internal teams on forecasting, backlog reduction, and operational readiness initiatives.
4. Performance Management & Reporting
ā¢ Track and report on key vendor KPIs (quality, timeliness, utilization, throughput).
ā¢ Analyze performance data to identify trends, highlight improvement areas, and communicate insights to internal stakeholders.
ā¢ Contribute to quarterly business reviews and strategic planning with vendor partners.
Experience:
ā¢ 7+ years of experience in Vendor Operations, Business Operations, or Program Management, ideally within a tech, AI, or content evaluation environment.
ā¢ Proven success managing and improving performance across multiple vendor partners.
ā¢ Strong analytical and problem-solving skills, with experience using data to drive operational decisions.
ā¢ Excellent communication and negotiation skills, with the ability to influence external partners and cross-functional stakeholders.
ā¢ Highly organized with a proactive approach to issue identification and resolution.
Skills:
ā¢ Human Evaluation
ā¢ Vendor Operations
ā¢ Content
ā¢ Root-cause
ā¢ Rater Operations
Education:
ā¢ Bachelor's degree.
About US Tech Solutions:
US Tech Solutions is a global staff augmentation firm providing a wide range of talent on-demand and total workforce solutions. To know more about US Tech Solutions, please visit www.ustechsolutions.com.
US Tech Solutions is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.
Apply tot his job
Apply To this Job