Lead Consultant, Senior Data Engineer (Databricks, AWS, MWAA)

Inviting applications for the role of Lead Consultant, Senior Data Engineer (Databricks, AWS, MWAA)

Seeking a Senior Data Engineer with strong Databricks, AWS, and MWAA (Managed Workflows for Apache Airflow) experience to support and execute a large-scale migration from Amazon Redshift to Databricks Lakehouse. The role is hands-on and critical to ensuring scalable, reliable, and well-orchestrated data pipelines.

Key Responsibilities

• Migrate data pipelines and workloads from Redshift to Databricks on AWS.

• Build and optimize ETL/ELT pipelines using Databricks (PySpark / Spark SQL).

• Design and manage workflow orchestration using MWAA (Apache Airflow).

• Refactor complex Redshift SQL into Databricks SQL / Spark.

• Implement Delta Lake for reliability, performance, and scalability.

• Integrate Databricks with AWS services (S3, IAM, Glue, Lambda, etc.).

• Ensure data quality, reconciliation, and post-migration validation.

• Monitor, troubleshoot, and optimize pipelines and Airflow DAGs.

• Collaborate with architects, BI teams, and stakeholders for smooth delivery.

Qualifications we seek in you!

Minimum Qualifications

• Strong hands-on experience with Databricks on AWS.

• Expertise in Apache Spark (PySpark, Spark SQL).

• Solid experience with MWAA (Apache Airflow) - DAG design, scheduling, monitoring, and error handling.

• Good understanding of Amazon Redshift and migration patterns.

• Proficiency in AWS services: S3, IAM, Glue (mandatory).

• Strong SQL and data warehousing fundamentals.

• Experience with performance tuning and cost optimization.

Preferred Qualifications

• Databricks Workflows, CI/CD for data pipelines.

• BI tool integration (Tableau / Power BI).

• Agile delivery experience and stakeholder interaction.