Senior Data Engineer

Note: - This is 6 Months Contractual role based on projects chances of extension is 90% for 12 months further with 100% remote opportunity.

We are looking for a highly experienced Azure Databricks Data Engineer with 8+ years of overall experience and strong hands-on expertise in Azure Databricks, PySpark, Databricks Asset Bundles (DAB), Azure DevOps, and CI/CD.

The ideal candidate will be responsible for designing and developing scalable data engineering solutions, implementing robust ETL/ELT pipelines, and automating Databricks deployments across multiple environments using DevOps best practices.

Key Responsibilities

• Design, develop, and maintain enterprise-scale data pipelines using Azure Databricks and PySpark.
• Develop scalable ETL/ELT solutions using PySpark, Spark SQL, and SQL.
• Build and optimize Delta Lake tables and Lakehouse data processing workflows.
• Implement and manage Databricks Asset Bundles (DAB) for packaging, configuration, and deployment of Databricks resources.
• Design and maintain Azure DevOps CI/CD pipelines for Databricks deployments.
• Automate deployment of Databricks notebooks, jobs, workflows, and configurations across Dev, QA, UAT, and Production environments.
• Integrate Databricks development with Git, GitHub, or Azure Repos.
• Implement environment-specific configurations and deployment strategies.
• Work with Azure Data Lake Storage Gen2, Azure Key Vault, Azure Data Factory, and other Azure services.
• Develop reusable PySpark frameworks and production-ready data processing solutions.
• Perform Spark performance tuning, including optimization of partitions, joins, caching, file sizes, and Delta Lake operations.
• Implement data quality, validation, logging, monitoring, and error-handling frameworks.
• Troubleshoot complex production issues and provide technical solutions.
• Collaborate with architects, data engineers, DevOps teams, and business stakeholders.
• Provide technical guidance and mentoring to junior and mid-level engineers.

Required Skills

Must Have

• 8+ years of overall experience in Data Engineering / Big Data / Cloud Data Engineering.
• Strong hands-on experience with Azure Databricks.
• Strong expertise in PySpark and Apache Spark.
• Hands-on experience with Databricks Asset Bundles (DAB).
• Strong experience with Azure DevOps and CI/CD pipelines.
• Experience automating Databricks deployments across multiple environments.
• Strong knowledge of Delta Lake and Lakehouse architecture.
• Strong SQL skills.
• Experience with Git / GitHub / Azure Repos.
• Experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
• Strong understanding of ETL/ELT concepts and modern data engineering practices.