Responsibilities :
Design, develop, and maintain scalable batch/stream data pipelines using Python and PySpark in distributed environments.
Implement efficient transformations, aggregations, and joins on large datasets while ensuring performance and cost optimization.
Write optimized SQL for data extraction, validation, and reconciliation across multiple sources.
Build reusable, testable modules and follow engineering best practices (code reviews, unit testing, documentation).
Troubleshoot production issues, perform root-cause analysis, and implement long-term fixes and monitoring improvements.
Collaborate with stakeholders to translate requirements into technical designs, delivery plans, and measurable outcomes.
Ensure data quality through validation checks, anomaly detection patterns, and consistent schema management.
Contribute to continuous improvement of development standards, performance benchmarks, and pipeline reliability.
Additional Responsibilities:
Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent practical experience).
5-9 years of hands-on experience in software development and/or data engineering roles.
Strong proficiency in Python with experience building production-grade applications or data workflows.
Strong proficiency in PySpark, including DataFrame APIs, optimization techniques, and distributed processing concepts.
Working knowledge of SQL for complex queries, data analysis, and validation.
Experience delivering reliable solutions with attention to performance, scalability, and maintainability.
