Senior Data Engineer - Databricks (Rotational Night Shift)

Position: Senior Data Engineer - Databricks (Rotational Night Shift)

Location: Pune/Nagpur/ Chennai/ Bangalore

Type of Employment: FTE

Purpose of the Position:

To design, build, and optimize & support scalable data pipelines and solutions using Databricks and related technologies, to make faster, data-driven decisions as part of our client data transformation journey. We are looking for data engineer with proficiency in data integration techniques, ETL processes and data pipeline architectures and well versed in Data Quality rules, principles and implementation.

Key Result Areas and Activities:

1. Data Pipeline Development & Support: Design, develop and optimize end to end batch and streaming data pipelines using Azure Databricks and Spark.
2. Data Architecture Implementation: Apply Medallion Architecture to structure data layers (raw, enriched, curated).
3. Data Quality & Governance: Ensure data accuracy, consistency, and governance using tools like Azure Purview and Unity Catalog.
4. Performance Optimization: Optimize Spark jobs, Delta Lake tables, and SQL queries for efficiency and cost-effectiveness.
5. Collaboration & Delivery: Work closely with analysts, architects, and business teams to deliver end-to-end data solutions.

Essential Skills:

• Hands-on experience with Databricks, Delta Lake, Data Factory.
• Proficiency in Python, PySpark, and SQL with strong query optimization skills.
• Deep understanding of Lakehouse architecture and Medallion design patterns.
• Experience building scalable ETL/ELT pipelines and data transformations.
• Experience with Git, CI/CD pipelines, and Agile methodologies.
• Rotational Shift including 2 weeks Night Shift.

Desirable Skills:

• Knowledge of data quality frameworks and monitoring practices.
• Experience with data visualization tools.
• Experience of working in managed services

Qualifications:

• Education: Likely a degree in Computer Science, Data Engineering, Information Systems, or a related field.
• Experience:
• Proven hands-on experience with Azure data stack (Databricks, Data Factory, Delta Lake).
• Experience in building scalable ETL/ELT pipelines.
• Familiarity with data governance and DevOps practices.

Qualities:

• Strong problem-solving and analytical skills
• Attention to detail and commitment to data quality
• Collaborative mindset and effective communication
• Proactive and self-driven
• Passion for learning and staying updated with emerging data technologies