Python - Data Processing

Responsibilities :

Key Responsibilities:

Design, develop, and maintain Python-based data processing workflows for structured and semi-structured datasets.

Build and optimize ETL processes to ingest, transform, validate, and publish data for downstream consumption.

Develop and manage data pipelines, ensuring reliability, scalability, and performance across workloads.

Write efficient SQL queries for data extraction, transformation, reconciliation, and reporting needs.

Implement data quality checks, validation rules, and monitoring to ensure accuracy and completeness of datasets.

Troubleshoot pipeline failures, identify root causes, and implement preventive fixes to reduce recurrence.

Collaborate with cross-functional teams to gather requirements, define data contracts, and deliver well-documented solutions.

Maintain clear technical documentation for pipeline logic, transformations, and operational runbooks.

Follow best practices for code quality, version control, testing, and peer reviews to ensure maintainable delivery. Minimum Qualifications:

BTECH, MTECH, MCA, or MSC in Computer Science, Information Technology, Data/Software Engineering, or a related field.

3â€5 years of experience in Python development focused on data processing and transformation use cases.

Hands-on experience building ETL workflows and maintaining production-grade data pipelines.

Strong SQL skills with experience in writing complex queries and optimizing performance.

Practical understanding of data processing concepts such as batching, partitioning, schema handling, and data validation.

Additional Responsibilities:

Preferred Qualifications:

Experience designing scalable pipeline patterns (incremental loads, CDC-style approaches, idempotent processing) and improving pipeline reliability.

Strong debugging skills with the ability to analyze data issues across multiple pipeline stages and resolve them efficiently.

Familiarity with orchestration and scheduling concepts for ETL jobs and dependency management.

Exposure to performance tuning for Python and SQL workloads, including memory-efficient processing and query optimization.

Proven ability to work with stakeholders to translate data requirements into robust, well-tested implementations.

Technical and Professional Requirements:

PYTHON DATA PROCESSING, DATAPIPELINE, ETL Pandas, NumPy, Apache Airflow, PySpark, Data Quality Frameworks