Location : Bangalore / Hyderabad / Pune
Experience : 5+ Type : Full-Time / Overview :
Responsibilities
We are looking for a highly skilled Azure Databricks Engineer with strong expertise in modern data engineering, big data processing, and distributed systems. The ideal candidate should have hands-on experience in building scalable ETL/ELT pipelines, optimizing Spark workloads, and working with enterprise grade Databricks Responsibilities :
- Design, develop, and maintain scalable data pipelines using Azure Databricks.
- Build and optimize ETL/ELT workflows for large-scale data processing.
- Work extensively with Apache Spark (PySpark & Spark SQL) for distributed data engineering.
- Implement Delta Lake architecture and optimize performance using Liquid Clustering and skew handling techniques.
- Develop reusable frameworks and automation solutions using Python.
- Optimize SQL queries, data models, and analytical workloads.
- Utilize Databricks Genie for engineering productivity and workflow acceleration.
- Collaborate with DevOps teams to implement CI/CD and Git-based deployment workflows.
- Work with orchestration and scheduling tools for automated pipeline execution.
- Ensure data quality, scalability, security, and reliability across Skills :
- 5+ years of experience in Data Engineering or related domains.
- Strong hands-on experience with Azure Databricks.
- Expertise in :
- Databricks Notebooks
- Delta Lake
- Job Orchestration
- Data Skewing
- Liquid Clustering
- Strong knowledge of Apache Spark :
- PySpark
- Spark SQL
- Spark Optimization Techniques
- Strong Python programming skills.
- Advanced SQL knowledge including :
- Complex Queries
- Performance Tuning
- Analytical Functions
- Experience with Databricks Genie.
- Strong understanding of CI/CD pipelines and Git workflows.
- Experience in data modeling and ETL/ELT pipeline design.
- Knowledge of automation frameworks and scheduling tools.
- Good understanding of distributed systems and big data Qualifications :
- Experience with cloud platforms like Microsoft Azure.
- Exposure to enterprise-scale data lake architecture.
- Strong problem-solving and analytical skills.
- Excellent communication and collaboration abilities.
(ref:hirist.tech)
