Big Data Engineer - Python/Scala, Spark, Hadoop

Job Description

Position: Data Engineer

Location: Arlington, VA

Job Description:

• Ability to easily move between business, data management, and technical teams; ability to quickly intuit the business use case and identify technical solutions to enable it

• Experience building robust and efficient data pipelines end-to-end with a strong focus on data quality

• High proficiency in using Python or Scala, Spark, Hadoop platforms & tools (Hive, Airflow, Ni Fi, Scoop), SQL to build Big Data products & platforms

• Experience in building and deploying production-level data-driven applications and data processing workflows/pipelines and/or

• Implementing machine learning systems at scale in Java, Scala, or Python and deliver analytics involving all phases like data ingestion, feature engineering, modeling, tuning, evaluating, monitoring, and presenting

• Cloud knowledge (Databricks or AWS ecosystem) is a plus but not required

Do

1. Managing the technical scope of the project in line with the requirements at all stages

a. Gather information from various sources (data warehouses, database, data integration and modelling) and interpret patterns and trends

b. Develop record management process and policies

c. Build and maintain relationships at all levels within the client base and understand their requirements.

d. Providing sales data, proposals, data insights and account reviews to the client base

e. Identify areas to increase efficiency and automation of processes

f. Set up and maintain automated data processes

g. Identify, evaluate and implement external services and tools to support data validation and cleansing.

h. Produce and track key performance indicators

2. Analyze the data sets and provide adequate information

a. Liaise with internal and external clients to fully understand data content

b. Design and carry out surveys and analyze survey data as per the customer requirement

c. Analyze and interpret complex data sets relating to customer’s business and prepare reports for internal and external audiences using business analytics reporting tools

d. Create data dashboards, graphs and visualization to showcase business performance and also provide sector and competitor benchmarking

e. Mine and analyze large datasets, draw valid inferences and present them successfully to management using a reporting tool

f. Develop predictive models and share insights with the clients as per their requirement

Deliver

No

Performance Parameter

Measure

Analyses data sets and provide relevant information to the client

No. Of automation done, On-Time Delivery, CSAT score, Zero customer escalation, data accuracy