Senior Apache Spark Technical Lead - Scala, Python

Job Summary

The Senior Technical Lead at HCL will be responsible for leading technical teams and projects related to apache spark, scala, and python. The role involves overseeing the design, development, and implementation of scalable and efficient solutions using these technologies.

Key Responsibilities

1. Lead technical teams in the design and implementation of solutions using apache spark, scala, and python
2. Provide technical expertise and guidance to team members in resolving complex technical issues
3. Collaborate with stakeholders to gather requirements and define project scope
4. Ensure adherence to best practices in coding, testing, and deployment processes
5. Conduct code reviews and performance optimization activities
6. Troubleshoot and debug technical issues to ensure seamless project delivery
7. Stay updated with the latest trends and advancements in apache spark, scala, and python technologies
8. Mentor team members and facilitate knowledge sharing within the team

Skill Requirements

• 8+ years of Engineering experience primarily in Data Engineering in an agile production environment
• High proficiency in using Python or Scala, Spark, Hadoop platforms & tools (Hive, Impala, Oozie, Airflow, Ni Fi, Scoop), SQL to build Big Data products & platforms
• Extensive experience with Databricks, Spark , Py Spark or Java + Py Sprak Processing engine.
• Proficiency in, at least, one modern programming language such as Python, Java or Scala
• Strong Computer Science fundamentals in object-oriented design, data structures, algorithm design, problem solving, and complexity analysis.
• Ability to easily move between business, data management, and technical teams; ability to quickly intuit the business use case and identify technical solutions to enable it
• Working Knowledge in Software Development engineering Paradigms along with Data Engineering.
• Relational Databases as well as NoSQL experience
• cloud technologies like Databricks/AWS/Azure ( Good to have)
• Basic Shell scripting and knowledge of Linux/Unix systems
• Experience in designing & developing software at scale
• Strong written and verbal English communication skills.
• Motivation, creativity, self-direction, and desire to thrive on small project teams
• Superior academic record with a degree in Computer Science or related technical field

Other Requirements

1.Relevant certifications in apache spark, scala, or python are a plus