Azure Senior Data Lead

Job Summary

Required Experience:

Bachelors’ degree in Information Technology, Computer Science or any Technology stream.5+ years of developing data pipelines & data infrastructure, ideally within a drug development or life sciences context.Demonstrated expertise in delivering large-scale information management technology solutions encompassing data integration and self-service analytics enablement.Experienced in software/data engineering practices (including versioning, release management, deployment of datasets, agile & related software tools).Ability to design, build and unit test applications on** Spark framework on Python**.Build Py Spark based applications for both** batch and streamingrequirements, which will require in-depth knowledge on Databricks**.Extensive working experience with SQL, including analysis and implementation of PL/pgSQL stored procedures.Experience working with MPP Data warehouses like** AWS Redshift**Experience working with storage frameworks like Delta Lake/ Iceberg Cloud-native, ideally AWS certified.

Key Responsibilities

1. Lead the end-to-end technical implementation of data projects using azure data factory, azure databricks, sql, oracle pl/sql, and python.

2. Design and develop efficient and reliable etl processes for large datasets.

3. Collaborate with cross functional teams to understand business requirements and translate them into technical solutions.

4. Optimize data workflows, troubleshoot issues, and ensure data quality and integrity.

5. Implement best practices for data security, governance, and compliance.

6. Provide technical guidance, mentoring, and support to junior team members.

7. Stay uptodate with the latest trends and technologies in data engineering and analytics.

Skill Requirements

1. Proficiency in azure data factory (adf) and azure databricks for data integration and processing.

2. Strong skills in writing complex sql queries for data manipulation and analysis.

3. Experience with oracle pl/sql for database development and management.

4. Proficient in python programming language for automation and scripting tasks.

5. Solid understanding of data warehousing concepts, etl processes, and data modeling.

6. Ability to work in a fast paced environment and manage multiple priorities effectively.

7. Excellent problem-solving skills and attention to detail.

8. Strong communication and interpersonal skills for effective collaboration with team members and stakeholders.

Other Requirements

1.Relevant certifications in Azure Data Factory, Azure Databricks, SQL, Oracle, or Python would be a plus.