Requisition Id: 1658692

Data Engineering Team Skill Requirements:

Exp: 6 to 12 Years.
Location: Hyderabad
Mode: Hybrid (3 days work from Office)

Data Engineer

Role

Build and maintain the future of Vanguard’s data platform on Databricks by designing and developing robust, scalable pipelines. Contribute to a validated, self-serve data ecosystem that supports analytics and business insights.

Base Skillsets

• Hands-on experience with Databricks (Spark, Delta Lake).

• Strong Python, Pyspark and SQL skills.

• Building and maintaining scalable ETL/ELT pipelines in AWS.

• Understanding of Medallion Architecture and data validation.

• Familiarity with orchestration tools (e.g., Airflow, Databricks Workflows).

• Experience and comfort with modelling with Kimball-style modelling (Star schemas).

Preferable Skillsets

• Experience with CI/CD and infrastructure-as-code (Terraform).

• Exposure to Delta Live Tables and Unity Catalog.

• Knowledge of Data Ops practices and data quality frameworks.

Responsibilities

• Design and deploy Databricks infrastructure using approved patterns.

• Develop and maintain ingestion pipelines via the Common Ingestion Framework.

• Collaborate with Product Managers and Business Analysts to validate data quality and ensure integrity.

• Define, document, and build tests within the Testing Framework defined within the program, including the Enterprise solution and the local, notebook-based process.

• Implement CI/CD for data pipelines and enforce best practices in testing and documentation.

• Monitor pipeline performance, troubleshoot issues, and optimize for cost and scalability.

• Document solutions and share knowledge with the engineering team.

• Support migration from existing infrastructure to Databricks.