Senior Apache Spark Technical Lead - Scala, Python

Job Summary

Role Summary

Support migration of data platforms to Databricks Lakehouse and assist in building scalable data pipelines for analytics.

Key Skills

• Hands-on with Databricks (PySpark, Spark SQL)

• Basic exposure to AWS/GCP (S3 or equivalent storage)

• Understanding of data lake / warehouse concepts

• Familiarity with Unity Catalog or data governance basics

• Exposure to data migration projects (preferred)

Responsibilities

• Develop and support data pipelines on Databricks

• Assist in data migration from legacy systems

• Work with S3/cloud storage for data ingestion

• Support table creation and optimization (Iceberg/Delta)

• Ensure data quality and performance

Key Responsibilities

1. Lead technical teams in the design and implementation of solutions using apache spark, Scala or Python
2. Provide technical expertise and guidance to team members in resolving complex technical issues
3. Collaborate with stakeholders to gather requirements and define project scope
4. Ensure adherence to best practices in coding, testing, and deployment processes
5. Conduct code reviews and performance optimization activities
6. Troubleshoot and debug technical issues to ensure seamless project delivery
7. Stay updated with the latest trends and advancements in Apache spark, scala, and python technologies
8. Mentor team members and facilitate knowledge sharing within the team

Skill Requirements

1. Strong proficiency in apache spark, scala, and python

2. Experience in leading technical teams and projects

3. Excellent problem-solving skills and ability to think critically

4. Solid understanding of software development life cycle

5. Good communication and interpersonal skills

6. Ability to work effectively in a collaborative team environment

7. Strong organizational and leadership abilities

8. Experience in performance tuning and optimization techniques

9. Knowledge of big data concepts and tools is a plus

Other Requirements

1.Relevant certifications in apache spark, scala, or python are a plus