Senior Data Engineer

Job Overview:

We are seeking a Senior Data Engineer with deep expertise in Databricks to design, develop, and optimize scalable data solutions. This role will be central to advancing our data engineering capabilities, with opportunities to work on cutting-edge technologies including Generative AI, advanced analytics, and cloud-native architectures.

You will collaborate with cross-functional teams, mentor junior engineers, and help shape the future of our data platform.

Key Responsibilities:

Data Architecture & Development: Lead the design, development, and optimization of scalable, secure, and high-performance data solutions in Databricks.

ETL/ELT Pipeline Engineering: Build and maintain robust pipelines, integrating Databricks with Azure Data Factory (or AWS Glue), ensuring reliability and efficiency.

Advanced Analytics & AI Integration: Implement machine learning models and Generative AI-powered solutions to deliver business innovation.

Collaboration: Partner with data scientists, analysts, and business teams to translate requirements into technical designs and deliverables.

Data Quality & Governance: Enforce best practices for data validation, governance, and security to ensure trust and compliance.

Technical Leadership: Mentor junior engineers, conduct code reviews, and foster a culture of continuous learning.

Innovation & Research: Stay current with the latest trends in Databricks, Azure Data Factory, cloud platforms, and AI/ML to recommend and implement improvements.

Qualifications:

Bachelor's or master's degree in computer science, Information Technology, or a related field.

5+ years in data engineering, with at least 3 years of hands-on experience in Databricks.

Proven expertise in:

Data modelling, Data Lakehouse architectures, and ELT/ETL processes

SQL and at least one programming language (Python or Scala)

Integration with Azure Data Factory or AWS Glue

Strong understanding of cloud platforms (Azure preferred) and containerization (Docker).

Excellent analytical, problem-solving, and communication skills.

Demonstrated experience mentoring or leading technical team

Preferred Skills:

Experience with Generative AI technologies and their applications.

Familiarity with other cloud platforms, such as AWS or GCP.

Knowledge of data governance frameworks and tools.