About the Role
Techylla is seeking an experienced Senior Data Engineer to design, build, and optimise enterprise-grade ELT pipelines for our cloud-based data platform. You will be a core contributor to our data infrastructure — ingesting data from multiple operational source systems and transforming it into reliable, governed, and performant datasets for analytics and reporting.
This role is central to our ELT pipeline operations — built primarily on Matillion — and you will also contribute to our Snowflake-based data warehouse initiatives.
If you take pride in writing clean, efficient, testable ELT code and want to help shape how a modern cloud data platform scales, this is the role for you.
Key Responsibilities
ELT Pipeline Development
- Design and develop orchestration and transformation jobs — leveraging Matillion — for batch and near-real-time ingestion across diverse source systems.
- Configure API-based (REST/SOAP), file-based (CSV, JSON, Parquet, XML), and database ingestion pipelines with robust error handling and alerting.
- Build and maintain incremental load and Change Data Capture (CDC) strategies to ensure low-latency, cost-efficient data freshness.
- Implement scheduling, dependency handling, and failure recovery mechanisms — pipelines must be production-grade, not just functional.
- Optimise ELT job performance and minimise cloud data warehouse compute costs through smart pipeline design, query push-down, and efficient scheduling.
Snowflake Platform Engineering
- Contribute to the design and build of Snowflake data warehouse — including database architecture, virtual warehouse sizing, and role-based access control.
- Develop Snowflake-native solutions: Snowpipe for continuous ingestion, Streams and Tasks for CDC and near-real-time processing, and Dynamic Tables where applicable.
- Implement Snowflake clustering keys, materialised views, and result caching strategies to optimise query performance at scale.
- Work with Snowflake's data sharing, data marketplace, and secure view features to support governed data access across teams.
- Manage Snowflake environment configuration including time travel, fail-safe settings, resource monitors, and cost governance controls.
Data Modelling & Platform Architecture
- Implement and maintain the ingestion framework including schema design, staging layers, and data zones (raw → cleansed → curated).
- Apply dimensional modelling, Data Vault 2.0, or medallion architecture patterns as appropriate to each use case, following agreed team standards.
- Manage platform configuration including security roles, data masking, column-level encryption, and access control policies.
- Support PII-compliant ingestion processes in line with GDPR and internal data governance policies.
- Contribute to data quality checks, testing frameworks, and ongoing monitoring of pipeline health and SLA adherence.
Collaboration & Standards
- Partner with business stakeholders, data analysts, and BI developers to translate requirements into scalable, documented data solutions.
- Produce clear technical documentation: pipeline designs, data dictionaries, architecture decision records (ADRs), and operational runbooks.
- Participate in code reviews, enforce development standards, and mentor junior engineers where applicable.
- Contribute to data governance practices including lineage tracking, metadata management, and data quality frameworks.
Must-Have Skills & Qualifications
- 3+ years of hands-on experience in Data Engineering roles delivering production pipelines at scale.
- 1+ years of hands-on ELT tool experience — Matillion strongly preferred — covering orchestration jobs, transformation jobs, component configuration, and scheduling. Production-level, not exploratory.
- 2+ years working with cloud-based data warehouse or lakehouse platforms — Snowflake strongly preferred; Databricks or Redshift also considered.
- Proven experience designing and delivering enterprise ELT pipelines covering API, file, and database sources.
- Strong SQL development skills: query optimisation, window functions, CTEs, execution plan analysis, and performance tuning.
- Solid understanding of incremental loading strategies, watermarking, and CDC patterns.
- Experience with file-based ingestion (CSV, JSON, Parquet, XML) and schema evolution handling.
- Ability to own work end-to-end — from requirement gathering through to documented, production-ready delivery.
Nice to Have
- Snowflake Advanced Features: Hands-on experience with Snowpipe, Streams, Tasks, Dynamic Tables, Snowflake Cortex, or data sharing features.
- Orchestration Tools: Familiarity with Apache Airflow, dbt (data build tool), Prefect, or similar pipeline orchestration and transformation frameworks.
- Data Modelling: Knowledge of Data Vault 2.0, dimensional modelling, or medallion (bronze-silver-gold) architecture patterns.
- Governance & Compliance: Experience with PII handling, GDPR-compliant ingestion, audit logging, and data lineage tooling (e.g., Alation, Collibra, or OpenLineage).
- ESG / Sustainability Data: Prior exposure to ESG reporting platforms or sustainability data workflows — auditability and traceability requirements are a key part of our platform.
- Matillion Certification: Matillion Certified Associate or Professional — or equivalent ELT tool certification.
- Snowflake Certification: SnowPro Core or SnowPro Advanced: Data Engineer certification.
- Version Control & CI/CD: Experience managing ELT pipeline code (e.g. Matillion jobs) in Git and contributing to CI/CD pipelines for data infrastructure.
What We Offer
- A dynamic, high-growth environment.
- Opportunity to work on impactful data projects with cross-functional visibility across the organisation.
- Competitive compensation aligned with market standards.
- Collaborative team culture that values clarity of communication and ownership of outcomes.
