Manual Evaluations Program Leader

1 week ago• San Francisco, CA (+1 more)
Sunnyvale, CA

Apply on company site

About Us

Uber is changing how people think about transportation, part of the logistical fabric of 600+ cities - giving people what they want when they want it.
Size: 10000+ employees

Industry: Technology

About the Role:

The Manual Evaluations Program Leader will own the end-to-end strategy, design and execution of human evaluations for Uber's GenAI-powered products, including conversational AI, voice AI, agent workflows and auto-evaluation systems. This role sits within the Global Digital Experience team, the operational arm of Uber's customer support tech organisation, and is a critical driver of quality, safety, and performance across Uber's next-generation AI solutions.

This leader will build and scale Uber's Manual Evaluation framework: defining methodologies, creating evaluation rubrics, ensuring annotation quality, and generating the insights that shape model tuning, product improvements, and release decisions. They will partner closely with Product, Engineering, Data Science and Product Ops to translate evaluation outcomes into clear technical and operational actions.

The role includes both strategic leadership and operational execution. The Program Leader will directly manage a team of three and indirectly oversee a distributed network of evaluators across global business sites. They will be responsible for setting the quality bar for evaluations, ensuring consistent delivery at scale, and driving continuous improvement of the evaluation pipeline.

The ideal candidate brings strong technical literacy in GenAI systems, exceptional program design and operational skills, and the ability to lead high-impact cross-functional initiatives. They are comfortable navigating ambiguity, building strong partnerships across Uber and influencing product direction through rigorous evaluation insights. This is a rare opportunity to play a leading role in one of Uber's most transformative technology programs and help shape the future of Uber's AI-driven experiences.

What the Candidate Will Do-Own the end-to-end strategy, design, and execution of Manual Evaluations for Uber's GenAI-powered products (chatbots, voice AI, automated workflows, autoeval systems)
-Develop and continuously improve evaluation methodologies, including rubrics, taxonomies, annotation guidelines, quality standards and success metrics

• Partner with Product, Engineering, Data Science and Product Operations to ensure human evaluations directly inform model tuning, safety improvements, product design changes, and release decisions, as well as scaled operations teams to delivery on time, at short notice and to a high quality standard
  -Lead evaluation projects across multiple AI products simultaneously, ensuring timelines, quality and delivery expectations are met
• Package insights into clear, actionable narratives and present them to cross-functional leaders, influencing product and operational strategy
  -Oversee a global manual evaluations operation, including direct management of a core team, indirect leadership of evaluators at multiple business sites and ongoing assessment of internal vs external resources to deliver the best evaluation outcomes
• Establish processes and tools that scale, including workflow optimization, evaluator training, QA systems and feedback loops.
• Serve as Uber's subject-matter expert in human evaluation for GenAI, staying current with best practices in safety testing, multimodal evaluation and human-in-the-loop systems.

Basic Qualifications- Bachelors degree in engineering or similar

• 5+ years of experience in program management, product operations, quality operations, research operations, or technical program leadership, ideally in a technology or AI-related environment.
• Experience with GenAI systems, LLM evaluation, model safety, failure pattern analysis, prompt evaluation, or AI product quality.
• Experience designing or running structured evaluation or quality frameworks, such as human labeling, annotation, audit workflows or manual review processes.
• Familiarity with evaluation methodologies(rubric design, taxonomies, annotation guidelines, reliability scoring, inter-rater agreement, etc.).
• Proven track record of managing teams, including coaching, performance management and resource planning.
• Strong project management abilities, with experience running multiple complex programs simultaneously.
• Proven experience managing outsourced teams to execute high-quality manual evaluation processes

Want more jobs like this?

Get jobs in San Francisco, CA delivered to your inbox every week.

Email Address

Send me The Muse newsletters for the best in career advice and job search tips.

Get jobs!
By signing up, you agree to our & .

Preferred Qualifications- Demonstrated ability to work cross-functionally with Product, Engineering, Data Science, and Operations teams.

• Knowledge of automated evaluation systems, LLM-as-judge frameworks, or hybrid human+machine evaluation pipelines.

• Background in service design, conversational AI, voice UX, or agent workflows.

• Strong analytical and problem-solving skills, with experience turning ambiguous data into clear insights.

• Excellent written and verbal communication skills, capable of translating technical evaluation outputs into business-relevant insights.

• Experience in global operations, including scaling teams, training processes, and quality management across regions.

• For San Francisco, CA-based roles: The base salary range for this role is USD**$162,000 per year**

• USD**$180,000 per year**.

You will be eligible to participate in Uber's bonus program, and may be offered an equity award & other types of comp. You will also be eligible for various benefits. More details can be found at the following link https://www.uber.com/careers/benefits.

Uber's mission is to reimagine the way the world moves for the better. Here, bold ideas create real-world impact, challenges drive growth, and speed fuels progress. What moves us, moves the world - let's move it forward, together.

Uber is proud to be an Equal Opportunity employer. All qualified applicants will receive consideration for employment without regard to sex, gender identity, sexual orientation, race, color, religion, national origin, disability, protected Veteran status, age, or any other characteristic protected by law. We also consider qualified applicants regardless of criminal histories, consistent with legal requirements. If you have a disability or special need that requires accommodation, please let us know by completing this form.

Offices continue to be central to collaboration and Uber's cultural identity. Unless formally approved to work fully remotely, Uber expects employees to spend at least half of their work time in their assigned office. For certain roles, such as those based at green-light hubs, employees are expected to be in-office for 100% of their time. Please speak with your recruiter to better understand in-office expectations for this role.

Client-provided location(s):San Francisco, CA, Sunnyvale, CA

Job ID: Uber-151923

Employment Type: FULL_TIME

Posted: 2026-01-20T00:30:34
Apply on company site

Perks and Benefits

Health and Wellness

• Health Insurance
• Health Reimbursement Account
• Dental Insurance
• Vision Insurance
• Life Insurance
• FSA With Employer Contribution
• Fitness Subsidies
• On-Site Gym
• Mental Health Benefits

Parental Benefits

Fertility Benefits:

Work Flexibility

• Flexible Work Hours
• Remote Work Opportunities
• Hybrid Work Opportunities

Office Life and Perks

• Casual Dress
• Pet-friendly Office
• Snacks
• Some Meals Provided
• On-Site Cafeteria

Vacation and Time Off

• Paid Vacation
• Unlimited Paid Time Off
• Paid Holidays
• Personal/Sick Days
• Sabbatical
• Volunteer Time Off

Financial and Retirement

• 401(K)
• Company Equity
• Performance Bonus

Professional Development

• Work Visa Sponsorship
• Associate or Rotational Training Program
• Promote From Within
• Mentor Program
• Access to Online Courses

Diversity and Inclusion

• Employee Resource Groups (ERG)
• Diversity, Equity, and Inclusion Program

Apply on company site

Similar Jobs

Suggested Searches

senior jobsUber jobsAll jobs

Search Additional Jobs

Manual Evaluations Program Leader Jobs in San Francisco, CAManual Evaluations Program Leader Jobs in Sunnyvale, CAJobs in San Francisco, CAJobs in Sunnyvale, CA

of Use](https://www.themuse.com/user/Popular Jobs

• Salaries

Get Involved

• For Employers
• The Muse Book: The New Rules of Work
• For Career Coaches
• Tell A Friend

Join The Conversation:

• Facebook
  
• LinkedIn
  
• Twitter
  
• Pinterest
  
• Instagram