Apache Spark Technical Lead - Scala, Python

Job Summary

Must have:

• ​4+ years’ experience in data engineering and/or software engineer experience in multiple environments

• 1+ years professional experience of utilising Apache Spark

• Experience using analytic tools such as Jupyter Notebooks, Apache Superset or equivalent

• Strong language capabilities with Python and Py Spark or SparkSQL

• Effective English communication skills

Nice to have / Willingness required

• Desirably experience or knowledge of Scala or Bash/Shell scripting

• Experience with Cloud (AWS/Azure/GCP)

• Knowledge with CI/CD, TDD, BDD

• Technical acumen in technologies associated with complex and deeply integrated systems eg: e2e domain, business logic and technology

• Experience in JAVA 8+;

• Experience in Java open-source frameworks, including developing using modern frameworks, e.g. Spring Boot;

• Experience in developing RESTful APIs and microservices-based system;

Key Responsibilities

1. Lead a team of developers in the design, implementation, and maintenance of data processing applications using apache spark, scala, and python.

2. Collaborate with cross functional teams to gather requirements, analyze data, and develop scalable solutions.

3. Troubleshoot and resolve technical issues related to data processing applications.

4. Implement best practices in coding, testing, and deployment processes.

5. Stay updated on the latest trends and advancements in apache spark, scala, and python technologies.

Skill Requirements

1. Proficiency in apache spark, scala, and python programming languages.

2. Strong understanding of data processing concepts and distributed computing.

3. Experience in leading and mentoring technical teams.

4. Excellent problem-solving skills and attention to detail.

5. Ability to work effectively in a fast paced and dynamic environment.

6. Good communication and interpersonal skills.

Other Requirements

1.Certification in apache spark, scala, or python is a plus.