Python+ETL+Pyspark

ETL Development

Design, develop, and maintain ETL pipelines using Python and Py Spark
Extract, transform, and load data from multiple structured and unstructured sources
Build reusable and scalable data processing frameworks
Ensure data quality, validation, and consistency

Py Spark / Big Data Processing

Develop and optimize Py Spark jobs for large-scale data processing

Work with Spark Data Frames and RDDs:

Implement transformations, aggregations, and joins in Spark
Optimize jobs for performance and scalability

Python Development

Develop backend logic and data processing scripts using Python
Write modular, reusable, and efficient code
Integrate APIs and automate workflows

Data Management & Integration:

Work with data lakes and warehouses (S3, HDFS, Redshift, Hive)
Handle file formats like Parquet, ORC, JSON, CSV
Perform data cleansing, enrichment, and transformation

Collaboration & Support:

Work with data engineers, analysts, and business stakeholders
Debug and troubleshoot ETL/data pipeline issues
Participate in Agile/Scrum ceremonies
Maintain documentation and coding standards

• Primary skills:Domain->Finacle-Core-Functional->Finacle-Core-WMS->Grand Master,Technology->ETL & Data Quality->ETL - Others,Technology->Machine Learning->Python

Core Skills:

2–5 years of experience in Python and ETL development
Hands-on experience with Py Spark (mandatory)
Strong understanding of data processing and pipelines
Solid knowledge of SQL and database concepts

Technical Skills:

Experience with Apache Spark ecosystem:

Good knowledge of data structures and algorithms (basic to intermediate)
Familiarity with Big Data technologies (Hadoop, Hive)
Experience with version control (Git)

Understanding of REST APIs and integrations:

Education: MCA,MSc,MTech,Bachelor of Engineering,BCA,BSc,BTech

• Preferred skills: Technology->AI-Data science->PYTHON,Technology->Big Data
• Data Processing->Py Spark,Technology->ETL & Data Quality->ETL - Others