Databricks Engineer

Responsibilities :

Roles & Responsibilities Design, develop, and maintain data pipelines using Databricks. Build ETL/ELT workflows for large-scale data processing. Develop and optimize data transformation logic using PySpark and Spark SQL. Implement and manage Delta Lake for reliable and efficient data storage. Work with batch and real-time data ingestion frameworks. Monitor and optimize data pipelines for performance and scalability. Collaborate with data analysts, data scientists, and business stakeholders. Ensure data quality, security, and governance standards are followed. Troubleshoot production issues and provide ongoing support.

Technical and Professional Requirements:

5-8 years of experience in Data Engineering and Big Data technologies.

Strong hands-on experience with Databricks, PySpark, and Spark SQL.

Experience with Delta Lake, Databricks Workflows, and Unity Catalog.

Proficiency in Python and SQL.

Experience with Azure/AWS/GCP cloud platforms.

Knowledge of Azure Data Factory (ADF), Apache Airflow, or similar orchestration tools.

Understanding of data warehousing concepts and ETL/ELT methodologies.

Experience with Kafka/Event Hub for streaming data processing is preferred.

Familiarity with Git, CI/CD pipelines, and DevOps practices.

Strong problem-solving, analytical, and communication skills.