Data Lead - Azure Data Factory (ADF), Databricks

Job Summary

Your Role:

As the Senior Data Engineer in the Data Foundation scope, you have in-depth knowledge of the technical details and how the products work, and you will act as an expert in the team responsible for the technical configuration, development and integration of products and platforms based on our tech stack, used by specific business processes across Op Cos, to better serve their customers.

You will provide expert guidance and ensure alignment with architecture and business objectives and ensure on-time delivery. You drive the end-to-end operations of the platforms, keeping them streamlined, structured, and within service level agreements.

You will collaborate closely with other cross functional teams for efficient development pipelines.

You will work directly with the Product Owner(s) and Product Architect(s), receiving pre-defined business requirements and translating them into technical specifications and services in line with overall engineering standards and roadmaps.

You will be expected to be implementing new features, deliver high quality code, follow the agile methodologies and be a team player. The most important part is to be part of the team effort towards value-driven outcomes and the successful completion of tasks. Your proactive approach will be key in maintaining comprehensive documentation and collaborating with your team members through offering help, raising question and actively taking part in all activities.

You will serve as a key contributor in refining and driving excellence in engineering practices to deliver high-quality solutions throughout the software development lifecycle in our Data landscape.

The role reports directly to the Data Engineering Lead

Key Responsibilities

Your responsibilities will include:

• Coach and mentor individual Data Engineers in designing, developing, and delivering scalable, reliable, and high performing big data solutions

• Coach and mentor the design, development, and maintenance of scalable data pipelines and ETL processes. Monitor and optimize data infrastructure performance, identifying and resolving bottlenecks and issues.

• Coach and mentor the team from a technical standpoint, and drive operational excellence, including code reviews, design reviews, testing, and deployment processes.

• Be an individual contributor (~60%) engineering the software products/solutions, jointly with the team

• Ensure that the team adheres to coding standards, best practices, and architectural guidelines, oversee team spirit and team performance, guide and mentor team members.

• Oversee the implementation of the technical architecture, solve immediate technical challenges.

• Ensure that the execution of Dev Sec Ops is in place in the team's daily work

• Inspire, advise, and drive the selection of development approach.

• Coordinate software development and address technical debt in the team

• Support onboarding, mentor, and develop top engineering talents, fostering a culture of learning,

• collaboration, and continuous improvement.

• Take Lead when needed in technical discussions with other teams/departments and oversees state-of-art quality of the stack.

• May be involved in cross-functional discussions, representing the domain in broader technical discussions across domains.

• Responsible for designing and improving processes that enhance efficiency and quality.

• Communication with Engineering Manager, Product Owner and Scrum Master to align on project / sprint goals, timeline and resource allocation.

Skill Requirements

Must have (all levels):

• Proficiency in programming languages such as Python, SQL, and experience with big data technologies like Massive Parallel Processing (MPP) and streaming.

• Experience with cloud platforms (e.g., AWS, Azure, GCP) and compute/data storage solutions (e.g., Databricks, BigQuery, Snowflake).

• Experience with CI/CD processes and tools, including Azure DevOps, Jenkins, and Git, to ensure smooth and efficient deployment of data solutions.

• Familiarity with APIs to push and pull data from data systems and Platforms.

• Familiarity with understanding software architecture High level Design document and translating them to developmental tasks.

• Expert in Pyspark

• Familiarity with Microsoft data stack such as Azure Data Factory, Azure Synapse, Databricks and Fabric / PowerBI.

• Apache Kafka or similar

• Data Modelling & Architecture

• ETL pipeline design

Nice to have:

• Logging and Monitoring using Azure / Databricks services

• Experience with machine learning and AI technologies

You are a good match if you have:

• 8+ years of experience in Data Engineering, with a strong understanding of data integration, ETL processes, and data warehousing.

• Hands-on experience and in-depth knowledge of the technologies listed as mandatory in the Technology Stack section

• Strong understanding and implementation of software development principles, coding standards, and modern architecture

• Familiarity with data governance and compliance standards.

• Hands-on experience in implementing and managing End-to-End Data Ops / Data Engineering projects in a team

• Proven ability to lead software development teams of engineers with varying experience and adapt to team sizes from small to large

• Experience in working in diverse projects with varying technologies, products, and systems

• Strong problem-solving skills and ability to make critical technical decisions

• Ability to guide / mentor other team members

• Effective communication and interpersonal skills, with the ability to collaborate with technical and non-technical stakeholders.

• Proven ability to demonstrate that can work independently and a self-starter

• Pragmatic, and collaborative team player

Other Requirements

• Excellent written and verbal English

• Other language required to perform job for the given Op Co(s)