Data Engineer

Role summary

We are seeking a Senior Data Engineer to design and build reliable batch and streaming data platforms. The role requires advanced Python and SQL, mandatory hands-on Apache Spark experience, strong data modelling and practical expertise with at least one cloud platform. The engineer will own ingestion, transformation, quality, orchestration, performance and operational reliability.

Key responsibilities

• Design and build scalable ETL/ELT pipelines for structured, semi-structured and streaming data.
• Develop distributed data-processing workloads using Apache Spark and Python/PySpark.
• Implement ingestion, transformation, validation, reconciliation and publishing workflows.
• Design cloud data lake, lakehouse and data-warehouse components.
• Implement incremental loading, change data capture, schema evolution and idempotent processing.
• Optimise Spark jobs, SQL queries, storage formats, partitioning and cloud resource utilisation.
• Build data-quality checks, monitoring, alerting, retry and recovery mechanisms.
• Implement orchestration and dependency management for production data pipelines.
• Collaborate with analytics, AI/ML and application teams to deliver trusted data products.
• Maintain documentation covering source-to-target mapping, lineage, operational runbooks and architecture.

Required qualifications and experience

• 5+ years of relevant experience in data engineering, big-data engineering or cloud data platforms.
• Strong Python and advanced SQL skills.
• Mandatory hands-on experience with Apache Spark, including performance tuning and distributed-processing concepts.
• Experience with at least one cloud platform: AWS, Microsoft Azure or Google Cloud Platform.
• Experience building and supporting production-grade data pipelines.
• Strong understanding of data modelling, data quality, orchestration and operational reliability.