Role Overview
We are looking for an experienced and results-driven Data Engineer to join our growing Data Engineering practice. The ideal candidate will be proficient in building scalable, high-performance data transformation pipelines using Snowflake, Azure Databricks, and Matillion, and will thrive in fast-paced, client-facing consulting environment. In this role, you will be instrumental in ingesting, transforming, and delivering high-quality data to enable data-driven decision-making across our clients organizations — owning your deliverables like a true Tech Doctor.
Key Responsibilities
- Design, configure, and optimize ingestion, transformation, and orchestration workflows using Matillion DPC where applicable.
- Design and implement scalable ELT pipelines using dbt on Snowflake, following industry-accepted best practices.
- Build and maintain data processing workloads on Azure Databricks (PySpark/Spark SQL), including notebooks, jobs, Delta Lake tables, and Lakehouse/medallion architectures.
- Build ingestion pipelines from various sources including relational databases, APIs, cloud storage, and flat files into Snowflake and Databricks (Delta Lake).
- Implement data modelling and transformation logic to support layered architecture (staging, intermediate, and mart layers, or medallion architecture) to enable reliable and reusable data assets.
- Leverage orchestration tools (e.g., Airflow, dbt Cloud, Azure Data Factory, or Databricks Workflows) to schedule and monitor data pipelines.
- Apply CI/CD and Git-based workflows for version-controlled deployments.
- Write well-documented, maintainable code using Git for version control and CI/CD processes.
Required Qualifications
- Mandatory: Hands-on experience with Azure Databricks — developing and deploying pipelines using PySpark/Spark SQL, Delta Lake, notebooks, clusters, and Databricks Workflows/Jobs in a production environment.
- Hands-on experience with Matillion Data Productivity Cloud (Matillion DPC) for data ingestion, transformation, or orchestration.
- Expert-level SQL and strong understanding of ELT principles; strong understanding of ELT patterns and data modelling (Kimball/Dimensional preferred).
- Experience with Git, CI/CD, and deployment workflows in a team setting.
- Familiarity with orchestrating workflows using tools like dbt Cloud, Airflow, Azure Data Factory, or Databricks Workflows.
- Comfort with ambiguity, competing priorities, and fast-changing client environments — we move fast and adapt faster.
- Passion for continuous learning and knowledge sharing (tech talks, meetups, internal upskilling).
