Apple is where individual imaginations gather together, committing to the values that lead to great work. Every new product we build, service we create, or Apple Store experience we deliver is the result of us making each other's ideas stronger. That happens because every one of us shares a belief that we can make something wonderful and share it with the world, changing lives for the better. It's the diversity of our people and their thinking that inspires the innovation that runs through everything we do. When we bring everybody in, we can do the best work of our lives. Here, you'll do more than join something - you'll add something.

We're looking for a Program Manager with a strong track record of building and leading effective evaluation programs demonstrating success leading via data. As a leader in the AIML Siri and Apple Intelligence evaluation team, you will lead initiatives in evaluation for foundation models and GenAI features powering Siri and Apple Intelligence experiences that are critical to Apple's future.

Description

You will work with top tier data scientists, engineers, research teams, and product teams across Apple to help ensure we deliver high-quality, safe, and beneficial AI-powered experiences that over 1 billion customers expect and love. This role requires technical depth in evaluation methodologies combined with strong program management expertise to drive comprehensive assessment of model capabilities, safety, helpfulness, and user experience quality.","responsibilities":"Lead the design and execution of evaluation programs for Siri and Apple Intelligence features, establishing comprehensive frameworks to assess model capabilities across multiple dimensions.

Oversee the development and implementation of evaluation infrastructure that scales with rapidly evolving model capabilities and provides real-time insights during training and deployment cycles

Drive consensus across multiple groups with varying opinions & needs.

Analyze evaluation results to identify patterns, failure modes, and opportunities for model improvement, translating complex findings into actionable insights for product and engineering teams

Foster a strong measurement-informed product development culture in SWE, influence partner organizations to develop rigor in measurement driven culture across Siri and Apple Intelligence team

Coordinate evaluation efforts during critical modeling and product milestones, ensuring comprehensive coverage, timely results, and alignment with product launch schedules

Provide clear, timely and objective communication on evaluation insights to executives across SWE and partner organizations to help inform critical product decisions and training strategies

Establish best practices and standards for evaluation development across the organization, enabling rapid iteration on evaluation design and supporting both automated and human-in-the-loop assessment approaches

Reflect on how the team operates, proactively propose and implement new ways to enable more effective processes & delivery of results that drive continuous improvement in evaluation quality

Preferred Qualifications

Master's or PhD degree in Statistics, Machine Learning, Computer Science, other Quantitative Sciences, or related field and equivalent experience

Experience with statistical analysis and drawing meaningful conclusions from large-scale evaluation datasets

Deep understanding of LLM capabilities, limitations, and safety considerations

Self-sufficient in analyzing and drawing conclusions about model quality, user experience, and product opportunity from raw and refined evaluation data

Player-coach capable of personally leading large evaluation initiatives while coaching team members along the way and mentoring team members to grow evaluation expertise

Minimum Qualifications

Bachelor's degree in Statistics, Business Intelligence, Computer Science, other Quantitative Sciences, or related field and equivalent experience

8+ years of experience in driving large scale program building machine learning powered products or analytics to support product development

5+ years of experience managing programs in AI powered product space, preferably experience in evaluation of ML/AI products

Ability to deal with ambiguities, drive disambiguation and clarities around evaluation methodologies, shepherd multiple teams to converge on rigorous measurement frameworks

Experience designing and implementing evaluation systems for machine learning models, particularly large language models or conversational AI systems

Program management skills including program structuring and managing multiple work streams interdependently across research, engineering, and product teams

Problem-solving skills with attention to details in identifying edge cases, failure modes, and capability gaps

Ability to communicate abstract ideas clearly, manage comprehensive yet succinct program status updates to all levels of audience, both verbally and in written forms

Proven adaptability and agility in making adjustments to program strategy and plan with evolving model capabilities and product decisions

Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant .

Pay & Benefits

At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $172,100 and $258,600, and your base pay will depend on your skills, qualifications, experience, and location.

Apple employees also have the opportunity to become an Apple shareholder through participation in Apple's discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple's Employee Stock Purchase Plan. You'll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses - including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits.

Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.