Job Summary
We are seeking a skilled
Databricks Data Engineer to design, build, and optimize scalable data pipelines and analytics solutions on the Databricks platform. The ideal candidate will have strong experience in
Lakeflow (Delta Live Tables), real-time streaming, DBSQL, and Data Intelligence frameworks, enabling efficient data processing, transformation, and insights generation.
Key Responsibilities
- Design and develop scalable data pipelines using Databricks Lakeflow (Delta Live Tables).
- Build and manage real-time and batch data processing systems using Structured Streaming.
- Develop and optimize DBSQL queries, dashboards, and data models for business intelligence and analytics.
- Implement data quality, governance, and monitoring within Lakehouse architecture.
- Work with large-scale distributed datasets using Apache Spark (PySpark/Scala).
- Integrate data from multiple sources (APIs, databases, cloud storage, streaming platforms).
- Collaborate with data scientists, analysts, and business teams to enable data-driven decision making.
- Ensure performance tuning and cost optimization of Databricks workloads.
- Implement Data Intelligence solutions including metadata management, lineage, and observability.
- Maintain CI/CD pipelines and follow best practices for deployment and version control.
Required Skills & Qualifications
- Strong experience with Databricks platform and Lakehouse architecture.
- Hands-on experience with Lakeflow / Delta Live Tables (DLT).
- Expertise in Structured Streaming and real-time data pipelines.
- Proficiency in DBSQL (Databricks SQL) for analytics and reporting.
- Strong programming skills in PySpark / Python / Scala.
- Experience with data modeling, ETL/ELT processes, and data warehousing concepts.
- Knowledge of cloud platforms such as AWS / Azure / GCP.
- Experience with data orchestration tools (Airflow, Azure Data Factory, etc.).
- Understanding of data governance, data quality, and security best practices.
Good to Have
- Experience with Delta Lake, Unity Catalog, and data lineage tools.
- Knowledge of Kafka / Event Hub / Kinesis for streaming ingestion.
- Familiarity with ML pipelines or MLOps workflows.
- Experience with BI tools like Power BI, Tableau, or Looker.
- Databricks certifications.
Soft Skills
- Strong problem-solving and analytical thinking.
- Good communication and stakeholder management skills.
- Ability to work in an agile and collaborative environment.
Skills: lakeflow,dbsql,data intelligence
