AGI Data Services strives to be best in class at acquiring, creating and ground-truth data, with the highest standards of privacy and trust, to power the best AI models on Earth.

We are seeking a Senior Software Development Engineer (Sr. SDE) who is passionate about Generative AI and has strong engineering fundamentals to own and accelerate the next generation of GenAI-powered tooling within AGI Data Services. The Sr. SDE will design, build, and maintain LLM-as-a-Judge evaluation pipelines that leverage large language models to assess data quality at scale — including judge architectures, evaluation rubrics, scoring models, and calibration mechanisms that align with the standards set by core scientist teams developing Amazon Nova models. The Sr. SDE will also design and build GenAI-powered workflow tools — such as conversational diagnostic agents, automated quality assessment systems, and guided remediation workflows — that streamline data collection and quality assurance processes, enabling cross-functional teams to rapidly identify issues, reduce resolution time, and continuously improve data throughput.

The Sr. SDE's work will directly improve Amazon Nova models. Our team has built a strong foundation of GenAI-powered engineering practices — this senior role will accelerate and scale that momentum. This role offers direct visibility to VP and SVP leadership.

Key job responsibilities
The Sr. SDE will own the LLM-as-a-Judge evaluation pipeline — designing, building, and scaling automated evaluation systems that leverage large language models to assess data quality. The Sr. SDE will architect judge pipelines, develop evaluation rubrics and scoring frameworks, build calibration and agreement mechanisms, and ensure judge outputs align with quality standards defined by core scientist teams.

The Sr. SDE will design and build GenAI-powered diagnostic and workflow tools — including conversational troubleshooting agents, automated quality assessment tools, guided remediation systems, and workflow copilots. The Sr. SDE will leverage and extend agent orchestration frameworks such as Lang Chain, Lang Graph, Amazon Bedrock Agents, or design custom orchestration layers tailored to AGI Data Services workflows.

The Sr. SDE will build upon the team's existing GenAI-forward practices — introducing advanced patterns for prompt engineering, RAG, agent orchestration, and LLM evaluation into production systems. The Sr. SDE will design and implement robust backend services, APIs, and data pipelines on AWS leveraging Amazon Bedrock, Sage Maker, Lambda, ECS/EKS, Step Functions, DynamoDB, Open Search, and S3.

The Sr. SDE will collaborate with Applied Scientists, Technical Program Managers, domain experts, and vendor teams — bridging technology, process, and operations.

A day in the life
The Sr. SDE will review LLM-as-a-Judge pipeline metrics — monitoring judge accuracy, calibration drift, and agreement rates — and collaborate with Applied Scientists to refine evaluation rubrics. The Sr. SDE will design new judge architectures, build and iterate on conversational troubleshooting agents, fine-tune prompt chains, and expand RAG knowledge bases. The Sr. SDE will dive deep into data quality anecdotes to find patterns and root causes, propose tooling solutions that automate manual processes, and share new GenAI integration patterns that build on existing team practices. The Sr. SDE will communicate impact and roadmaps to cross-functional partners and VP leadership.

Basic Qualifications

• 5+ years of non-internship professional software development experience
• 5+ years of programming with at least one software programming language experience
• 5+ years of leading design or architecture (design patterns, reliability and scaling) of new and existing systems experience
• Experience as a mentor, tech lead or leading an engineering team

Preferred Qualifications

• 5+ years of full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations experience
• Bachelor's degree in computer science or equivalent

Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.

Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.

The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location. Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at https://amazon.jobs/en/benefits.

USA, WA, BELLEVUE - 168,100.00 - 227,400.00 USD annually