This is a data engineer position - a programmer responsible for the design, development implementation and maintenance of data flow channels and data processing systems that support the collection, storage, batch and real-time processing, and analysis of information in a scalable, repeatable, and secure manner in coordination with the Data & Analytics team.

The overall objective is defining optimal solutions to data collection, processing, and warehousing. Must be a Spark Java development expertise in big data processing, Python and Apache spark particularly within banking & finance domain.  He/She designs, codes and tests data systems and works on implementing those into the internal infrastructure.

Responsibilities:

• Ensuring high quality software development, with complete documentation and traceability
• Develop and optimize scalable Spark Java-based data pipelines for processing and analyzing large scale financial data
• Design and implement distributed computing solutions for risk modeling, pricing and regulatory compliance
• Ensure efficient data storage and retrieval using Big Data
• Implement best practices for spark performance tuning including partition, caching and memory management
• Maintain high code quality through testing, CI/CD pipelines and version control (Git, Jenkins)
• Work on batch processing frameworks for Market risk analytics
• Promoting unit/functional testing and code inspection processes
• Work with business stakeholders and Business Analysts to understand the requirements
• Work with other data scientists to understand and interpret complex datasets

Qualifications:

• 5- 8 Years of experience in working in data eco systems.
• 4-5 years of hands-on experience in Hadoop, Scala, Java, Spark, Hive, Kafka, Impala, Unix Scripting and other Big data frameworks.
• 3+ years of experience with relational SQL and NoSQL databases: Oracle, MongoDB, HBase
• Strong proficiency in Python and Spark Java with knowledge of core spark concepts (RDDs, Dataframes, Spark Streaming, etc) and Scala and SQL
• Data Integration, Migration & Large Scale ETL experience (Common ETL platforms such as PySpark/DataStage/AbInitio etc.) - ETL design & build, handling, reconciliation and normalization
• Data Modeling experience (OLAP, OLTP, Logical/Physical Modeling, Normalization, knowledge on performance tuning)
• Experienced in working with large and multiple datasets and data warehouses
• Experience building and optimizing ‘big data’ data pipelines, architectures, and datasets.
• Strong analytic skills and experience working with unstructured datasets
• Ability to effectively use complex analytical, interpretive, and problem-solving techniques
• Experience with Confluent Kafka, Redhat JBPM, CI/CD build pipelines and toolchain – Git, BitBucket, Jira
• Experience with external cloud platform such as OpenShift, AWS & GCP
• Experience with container technologies (Docker, Pivotal Cloud Foundry) and supporting frameworks (Kubernetes, OpenShift, Mesos)
• Experienced in integrating search solution with middleware & distributed messaging - Kafka
• Highly effective interpersonal and communication skills with tech/non-tech stakeholders.
• Experienced in software development life cycle and good problem-solving skills.
• Excellent problem-solving skills and strong mathematical and analytical mindset
• Ability to work in a fast-paced financial environment

Education:

• Bachelor’s/University degree or equivalent experience in computer science, engineering, or similar domain

Technical Skills: ​

• Very proficient in Scala & Python
• Advance SQL knowledge
• Very good and fast debugging skills
• Versioning - Github (or Bitbucket )
• CICD pipeline - RLM / UDeploy / Harnes
• IDE - Intilij and vsCode
• Big Data - HDFS / Hive / Impala
• Unix - Scripting skills
• Database - Hive / Hbase / Snowflake / Mongo