Director AI Evaluation

• Reports to the VP of AI; directly career-manages the data science line and matrix-manages the Senior Analysts, AI Evaluation.
• Determines the quality standard for any high-value AI initiative at Geisinger — internally built or vendor-provided — from design through production.
• Holds bought systems to the same standard as built ones, generating local evidence on whether a tool works here for Geisinger's clinicians and patients rather than accepting vendor aggregate or cherry-picked results.
• Owns the methodology that holds initiatives to that standard: pre-production validation and live production monitoring.
• Owns the health of the data science team — attracting and retaining strong technical talent, developing careers, and keeping the bench deep, engaged, and growing — and leads and develops the evaluation team alongside it.
• Provides hands-on technical guidance to program teams as they design validation studies, equity audits, monitoring plans, and escalation playbooks.
• Owns the evaluation toolkit and reusable playbooks and templates that let each new program move faster than the last.
• Translates program-specific failure modes into concrete, measurable production-monitoring metrics; defines what is measured and how, while the AI Platform team builds the backend.
• Tracks AI System Performance — the single most important accuracy indicator for each system, against thresholds set to clinical tolerance.
• Tracks User Adoption — engagement, override rates, and time-to-action — distinguishing genuine workflow misalignment and alarm fatigue from poor predictive value.
• Connects each AI to the Outcome it was deployed to improve (mortality, time-to-treatment, boarding time, denial rate, cost per case) against a pre-launch baseline over a use-case-appropriate horizon, holding both tangible returns and harder-to-quantify value in view.
• Monitors Equity — the maximum performance gap on the Pillar 1 metric across the subgroups that matter for the initiative, so disparate impact surfaces early.

Work is typically performed in an office environment. Accountable for satisfying all job specific obligations and complying with all organization policies and procedures. The specific statements in this profile are not intended to be all-inclusive. They represent typical elements considered necessary to successfully perform the job.