Job Role: Senior Python Developer (PySpark)
Experience: 5–7 Years
Location: Chennai
Key Responsibilities
- Design & develop robust ingestion, transformation, and enrichment pipelines with Python, PySpark, and SQL.
- Write and optimize complex SQL queries, analytical UDFs, and window functions for data aggregation and reporting.
- Collaborate with CEFS data architects, data scientists, and business analysts to translate functional requirements into technical specifications.
- Unit‑test, integrate‑test, and review code.
- Maintain CI/CD pipelines (Git, Jenkins, Docker) for automated build, test, and deployment of jobs.
- Monitor production workloads and troubleshoot performance bottlenecks, memory issues, and job failures.
- Document data lineage, pipeline design, and operational run‑books in Confluence/SharePoint.
- Keep up to date with latest technologies and trends and provide input, expertise and recommendations.
Required Skills
- 5–7 years of experience in Python Development with Spark.
- Strong hands-on experience with Python, PySpark, SQL, and ETL development.
- PySpark Expertise in DataFrames, Spark SQL, Structured Streaming & performance tuning.
- Hands-on experience with NumPy, pandas, Python Frameworks, Restful APIs, MS-SQL /Oracle.
- Strong SQL skills including query optimization, stored procedures, and data modeling.
- Experience with CI/CD tools such as Git, Jenkins, Bitbucket, and Docker.
- Knowledge of Linux/Unix environments and shell scripting.
- Familiarity with Agile, Scrum, and DevOps methodologies.
- Good to have exposure with Flask /REST API development.
- Exposure to Data Analytics, AI/ML, or Big Data platforms.
- Strong problem-solving, communication, and mentoring skills
