Duties
- We are looking for a hands-on Data Platform Engineer to support the DCEE Data & Insights Platform, a cloud-based data platform enabling secure, high-quality, near real-time reporting, analytics and data product delivery.
- The role requires experience across Kafka ingestion, Snowflake data engineering, dbt transformations, Data Vault 2.0 modelling, Kubernetes/EKS deployments, GitOps, CI/CD, monitoring and production troubleshooting.
- The engineer will support both the Kafka repository for ingestion and connector operations, and the Snowflake repository for data product development, dbt models, masking, grants, streams, tasks and data quality support.
- DIP follows Data Mesh, Event Driven Architecture and Data Vault 2.0 principles with a core stack including Confluent Kafka, Kafka Connectors, EKS, Snowflake, Python, GitHub Actions and ArgoCD.
Skills
Must have skills: AWS, GitHub, Confluent Kafka Cloud, Kafka ingestion, Snowflake data engineering, dbt transformations, Data Vault 2.0 modelling, Kubernetes/EKS deployments, CI/CD
Other Skills
- Kafka, CDC & Ingestion: Apache Kafka and Confluent Kafka Cloud, Kafka Connect, Debezium CDC, Snowflake Sink Connector, topic/offset management, incremental ingestion, schema change handling.
- Snowflake Data Engineering: Snowflake SQL, databases, schemas, warehouses, roles, grants, Streams, Tasks, Information Schema, RDL/RDV/ID/BDV layer development.
- Dbt, Data Vault & Data Products: dbt Core, dbt-snowflake, SQL/YAML models, dbt tests and documentation, custom macros, Data Vault 2.0, Hubs, Links, Satellites and Data Mesh concepts.
- Cloud, Kubernetes & DevOps: AWS, EKS, Kubernetes, kubectl, ArgoCD, GitHub, GitHub Actions, CI/CD troubleshooting, Docker/JFrog/SonarQube basics.
- Monitoring & Production Support: Splunk, Dynatrace, Tableau alerts, Kafka Connect REST APIs, connector health checks, Snowflake task monitoring, incident troubleshooting and root cause analysis.
- Security, Access & Governance: Snowflake masking policies, role-based access, grants, least privilege practices, GitHub Secrets, AWS Secrets Manager, HashiCorp Vault and secret rotation.
- Programming & Source Systems: Python, Snowpark Python, MySQL, Oracle, PostgreSQL, RDS connectivity, source database CDC concepts and data validation/reconciliation.
Preferred Experience
Experience supporting production data platforms with near real-time ingestion, hands-on exposure to both Kafka and Snowflake repositories, and ability to troubleshoot issues across source systems, connectors, Kafka, Snowflake, dbt and CI/CD pipelines.
Day To Day Work (same As Preferred Experience)
- Experience supporting production data platforms with near real-time ingestion
- Hands-on experience with both Kafka ingestion and Snowflake transformation layers
- Experience working with GitOps-based deployments using ArgoCD
- Experience building data products using dbt and Snowflake
- Familiarity with Data Vault 2.0 and Data Mesh operating models
- Ability to debug issues across source database, Kafka, connector, Snowflake and dbt layers
- Strong ownership mindset for platform reliability, data quality and operational support
Education
Bachelor's degree in quantitative field like Computer Science, Engineering, Statistics, Mathematics or related field required. Advanced degree is a strong plus
