Data Engineer II - Confluent

About the Role:

As a Data Engineer II of Confluent, an IBM company, Data team you will own meaningful slices of our data platform end-to-end -- from ingestion through transformation to the data products that power decision-making across the company. You will be part of Confluent R&D team and work closely with various business stakeholder like Prod&Eng, Marketing, Fields Ops, Sales etc to address their analytical needs. You would be working closely with the overall IBM CDO and CIO team to enable the wider organisation with data needs from the Confluent ecosystem. You'll design and operate batch and real-time pipelines, partner closely with Data Scientists, Analysts, and business stakeholders, and raise the bar on craftsmanship, reliability, and developer productivity for the team.

You'll work in a modern, AI-augmented engineering environment: shipping faster with Claude Code and other AI coding assistants, building AI-powered internal tools (natural-language-to-SQL, automated data quality, lineage, anomaly detection), and bringing GenAI thoughtfully into data workflows. We expect you to take significant projects from ambiguous problem statement to delivered outcome, mentor newer engineers, and represent the team confidently with cross-functional partners.

What You Will Do:

• Design, build, and operate efficient, reliable, and well-documented data pipelines across batch and streaming systems -- from source ingestion through the warehouse to consumer-facing data products.

• Own the data models, SLAs, and quality contracts for the domains you cover; treat documentation, testing, and observability as first-class deliverables.

• Improve the data platform itself: introduce or extend tooling, templates, CI/CD, testing frameworks, and monitoring that make the entire team faster and more reliable.

• Adopt and champion AI-assisted engineering -- use Claude Code and similar tools as part of your daily workflow, build internal AI-powered utilities (text-to-SQL on the semantic layer, automated PR review, log triage, data discovery), and evaluate where LLMs add durable value vs. hype.

• Partner directly with Data Scientists, Analysts, and business stakeholders to translate ambiguous requirements into well-scoped deliverables; communicate trade-offs, anticipate push-back, and drive alignment without needing escalation.

• Mentor interns, new hires, and L2 engineers -- through onboarding, code review, design feedback, and pairing.

• Contribute to cloud infrastructure and governance: IAM, service accounts, secrets management, cost optimization, and Terraform-managed GCP / Confluent Cloud resources.

• Drive incident response and root-cause analysis for the pipelines you own; close the loop with durable fixes, runbooks, and prevention.