Data Engineer - Databricks

Infosys Brazil is looking for professionals with the profile of Data Engineer-Databricks to join our team.

Main activities and responsibilities:

• Lead the design and implementation of advanced data solutions on Databricks.
• Architect scalable, secure, and efficient data platforms using Py Spark, Python, and SQL.
• Optimize ETL processes and data workflows for performance and reliability.
• Collaborate with data engineers, data scientists, and business stakeholders to understand requirements and deliver robust solutions.
• Set up and manage CI/CD pipelines and enforce data governance standards.
• Monitor and troubleshoot data pipelines to ensure data quality and integrity.
• Mentor junior team members and foster a culture of continuous learning.
• Stay updated with emerging technologies and promote innovation in data practices.

Required Qualifications:

• Advanced English.

Advanced experience in Databricks:

• Practice in designing and building Databricks data pipelines using Py Spark, SQL; Databricks workflows for orchestration and scheduling; Azure/AWS cloud storage services; understanding of different types of Databricks clusters.
• SQL proficiency:
• Proficiency in writing complex SQL queries using complex joins, window functions, CTEs.
• Transpose, Joins, Window Functions, Deduplication.

Py Spark Proficiency:

• Hands on experience with joins, filters, coalesce, group By, window operations.
• Data Frame API concepts and Delta Lake fundamentals.
• Data modeling:
• Experience in data modeling to design star schema.
• Experience in Enterprise Data warehousing projects.
• Normalization and denormalization approaches.

Type of work: CLT, Flexible hours.
Benefits package: Transport vouchers, meal vouchers, childcare assistance, life insurance, funeral assistance, medical and dental insurance.
Career: Global work on national and international clients and/or projects, possibility of international career movement.