Overview

Microsoft is a company where passionate innovators come to collaborate, envision what can be and take their careers further. This is a world of more possibilities, more innovation, more openness, and the sky is the limit thinking in a cloud-enabled world.

Microsoft’s Azure Data engineering team is leading the transformation of analytics in the world of data with products like databases, data integration, big data analytics, messaging & real-time analytics, and business intelligence. The products our portfolio include Microsoft Fabric, Azure SQL DB, Azure Cosmos DB, Azure PostgreSQL, Azure Data Factory, Azure Synapse Analytics, Azure Service Bus, Azure Event Grid, and Power BI. Our mission is to build the data platform for the age of AI, powering a new class of data-first applications and driving a data culture.

​​Within Azure Data, the messaging and real-time analytics team provides comprehensive solutions and a robust platform that enables users to ingest high granularity signals (real-time & observability) and complex data, converting those into a competitive advantage in real-time for both end users and modern applications.  ​

​​The Real-Time Intelligence team is hiring a Senior AI Engineer to build and scale the evals platform that underpins our real-time, agentic solutions on live operational data. You will focus on creating human and LLM-as-judge pipelines, and telemetry that measure how well our autonomous agents, anomaly detection, and decisioning systems perform in the real world. In this role, you’ll own end-to-end implementation of eval pipelines and tooling, integrate them into CI/CD and production workflows, and partner closely with product and platform teams so that every AI feature is measurable, debuggable, and continuously improving. Rigorous evaluation is the most critical ingredient in building reliable agents, this is a highly leveraged, key role that will directly shape how we design, ship, and operate real-time AI agentic products.

​​

We do not just value differences or different perspectives. We seek them out and invite them in so we can tap into the collective power of everyone in the company. As a result, our customers are better served

Responsibilities:

​​- Build and evolve the Real-Time Intelligence evaluations platform: implement offline and online eval pipelines, including golden datasets, human review workflows, and LLM-as-judge / auto-raters for agents, anomaly detectors, and decisioning systems.

• Instrument agentic solutions for observability by wiring up telemetry, tracing, structured logging, and dashboards so quality, safety, latency, and cost are easy to monitor and debug.
• Integrate evals into the development lifecycle by connecting pipelines to CI/CD, canary and A/B experiments, and phased rollouts, making it simple for partner teams to run and interpret evaluations.
• Collaborate and mentor across product, research, and engineering teams, sharing best practices on eval design, LLM-as-judge usage, and Responsible AI, and providing code reviews and guidance that raise the bar for the AI features.​

Embody our culture and values

Qualifications Required/Minimum Qualifications:

​​​​- Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience.

Other Requirements

Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check:

This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Preferred/Additional Qualifications

​​​​- Master's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience.

• Demonstrated experience shipping LLM-powered features or agentic systems in production (e.g., copilots, assistants, workflow agents, tool-calling systems).

• Experience with at least one major cloud platform (Azure, AWS, or GCP) and modern infrastructure practices (microservices, containers, orchestration, CI/CD). ​ ​

• Proven ability to define architectures, make trade-offs, and lead multi-team projects from design through launch.

• Background in evaluation of ML/LLM systems: designing test sets, rubrics, human and auto-rater evaluation pipelines.

• Excellent communication skills and the ability to explain complex trade-offs to both technical and non-technical stakeholders. ​ ​

​​​​- Master's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience.

• Proven ability to define architectures, make trade-offs, and lead multi-team projects from design through launch.
• Background in evaluation of ML/LLM systems: designing test sets, rubrics, human and auto-rater evaluation pipelines.
• Excellent communication skills and the ability to explain complex trade-offs to both technical and non-technical stakeholders. ​ ​

#azdat

#azuredata

Software Engineering IC4 - The typical base pay range for this role across the U.S. is USD $119,800 - $234,700 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $158,400 - $258,000 per year.

Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
https://careers.microsoft.com/us/en/us-corporate-pay

This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.

Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.