Locations: Kolkata
Experience: 8+ Years
Role: Data Engineer/Developer
Must-Have:
Python
Spark
Databricks
AWS (S3, Glue, Airflow, CloudWatch, Lambda)
Responsibility:
- Master Databricks tools (job creation, cluster, notebook) and be able to query efficiently with SQL
- Maintain the platform in operational condition in production (analyze and correct incidents and defects)
- Development of Python data ingestion and transformation jobs with Spark on large volumes of data
- Provide a long-term vision, both operationally and in terms of data platform strategy
- Support and promote best practices
- Participate in technical and functional design workshops
- Write and update technical documentation
Methodologies:
- CI/CD avec Gitlab
- JIRA / Confluence
- Scrum
