Databricks Engineer

About The Role

We are looking for an experienced Databricks Engineer with 12+ years of overall IT experience to join our team on a long-term contract. The ideal candidate should have deep expertise in Databricks, Apache Spark, PySpark, Python, SQL, and cloud platforms (AWS, Azure, or GCP), along with a proven track record of delivering enterprise-scale data engineering solutions.

This is a customer-facing role requiring strong technical leadership, solution architecture capabilities, and hands-on experience building scalable cloud-based data platforms.

Mandatory Certification

• Databricks Certified Data Engineer Professional (Mandatory)
  
  

Key Responsibilities

• Lead end-to-end Databricks implementation projects from solution design through deployment and production support.
• Design, develop, and optimize scalable data pipelines using Databricks, Apache Spark, and PySpark.
• Build cloud-native data engineering solutions using AWS, Microsoft Azure, or Google Cloud Platform (GCP).
• Design and implement modern Lakehouse architectures for enterprise-scale data platforms.
• Translate business requirements into scalable, high-performance technical solutions.
• Integrate Databricks solutions with enterprise applications, APIs, and third-party systems.
• Configure, monitor, troubleshoot, and optimize Databricks environments for performance, reliability, and scalability.
• Implement CI/CD pipelines and DevOps best practices for data engineering workflows.
• Collaborate with architects, business stakeholders, and cross-functional teams to deliver high-quality solutions.
• Provide technical leadership, mentor engineering teams, and promote best practices.
• Participate in customer workshops, architecture discussions, and technical consulting sessions.
• Ensure solution quality, security, governance, and operational excellence throughout project delivery.
  
  

Required Skills

• Databricks
• Apache Spark
• PySpark
• Python
• SQL
• Data Engineering
• ETL/ELT Pipeline Development
• Lakehouse Architecture
• AWS / Microsoft Azure / Google Cloud Platform (GCP)
• Solution Architecture
• Systems Integration
• CI/CD Pipelines
• Performance Optimization
• Data Modeling & Data Warehousing
• MLOps
• Git & DevOps Practices
  
  

Required Experience

• 12+ years of overall IT experience.
• Hands-on expertise with Databricks and modern data engineering platforms.
• Experience delivering 10+ end-to-end Databricks implementation projects.
• Strong experience with distributed data processing using Apache Spark and PySpark.
• Proven experience designing and implementing enterprise-scale cloud-based data platforms.
• Strong understanding of Lakehouse Architecture, data integration, governance, and data modeling.
• Experience implementing CI/CD pipelines and deployment automation.
• Excellent troubleshooting, debugging, and performance tuning skills.
• Strong client-facing consulting and stakeholder management experience.
  
  

Preferred Qualifications

• Experience with Delta Lake, Unity Catalog, MLflow, and Databricks Workflows.
• Knowledge of modern data architecture patterns and cloud-native services.
• Experience working in Agile/Scrum environments.
• AWS, Azure, or GCP certifications are an added advantage.
  
  

Soft Skills

• Excellent communication and presentation skills.
• Strong analytical and problem-solving abilities.
• Customer-first mindset with strong consulting skills.
• Ability to manage multiple priorities in a fast-paced environment.
• Leadership, ownership, and mentoring capabilities.