Senior Data Engineer / Platform Re-Engineering Lead – Azure Synapse & Databricks Migration
Company: Claidroid Technologies Pvt. Ltd.
Location: Thiruvananthapuram, Kerala / Pune, Maharashtra
Work Mode: Hybrid
Experience: 10+ years
Employment Type: Full-time
About Claidroid Technologies
Claidroid Technologies Pvt. Ltd. is hiring an experienced Senior Data Engineer / Platform Re-Engineering Lead – Azure Synapse & Databricks Migration to join our growing team.
At Claidroid Technologies, our name is derived from Cloud, AI, and Automation — the three pillars that define our innovation-driven approach to digital transformation.
Claidroid Technologies is a global digital transformation and technology services partner built by IT professionals with over three decades of industry experience, with a presence across India, Europe, and the USA. We help enterprises modernize platforms, improve operational efficiency, strengthen governance, and accelerate business outcomes. We work with leading enterprises in regulated industries, delivering scalable, high-quality technology solutions with a strong focus on governance, quality assurance, and business alignment.
Role Overview
As a Senior Data Engineer / Platform Re-Engineering Lead, you will lead the re-engineering of an existing enterprise data platform built on Azure Synapse Analytics and drive the future migration of validated workloads to Databricks.
This is not a greenfield implementation role. The role requires deep technical seniority to audit, understand, validate, and improve a complex end-to-end data architecture spanning source ingestion, transformation, governance, and consumption layers.
You will be responsible for reverse-engineering existing implementations, assessing their correctness, identifying gaps and defects, and owning the technical migration strategy to a modern Databricks Lakehouse platform.
You will work closely with senior engineers, business stakeholders, data governance teams, and architecture teams to validate embedded business logic, preserve governance and control framework semantics, and ensure platform transformation is delivered with quality, accuracy, and business continuity.
Key Responsibilities
- Lead the technical assessment and re-engineering of an existing enterprise data platform across all layers, from source ingestion through to data consumption.
- Reverse-engineer, document, and validate existing pipeline logic, data models, transformation frameworks, and data governance controls.
- Identify gaps, defects, and technical debt across the platform and remediate incorrect or sub-optimal implementations.
- Ensure correctness of data processing patterns, including change data capture, slowly changing dimensions, deduplication, soft/hard deletes, retroactive change processing, and business reconciliation.
- Design and implement target-state architectures aligned to modern lakehouse principles while ensuring feature parity and business logic fidelity during transition.
- Manage platform evolution initiatives, including parallel-run phases where multiple implementations operate simultaneously.
- Validate output consistency before cutover and contribute to legacy component decommissioning planning.
- Define and execute migration strategies for existing workloads from Azure Synapse to Databricks.
- Re-implement ingestion, transformation, and orchestration pipelines on target platforms while maintaining audit, quality, reconciliation, and governance standards.
- Collaborate with business, data governance, and architecture stakeholders to validate embedded business rules and data quality requirements.
- Provide technical leadership across re-engineering and migration workstreams.
Key Requirements
- 10+ years of experience in Data Engineering, with significant platform migration or re-engineering experience.
- Proven experience in auditing and taking ownership of existing, complex enterprise data platforms — not just building from scratch.
- Strong hands-on experience with Azure Synapse Analytics, including Pipelines, Spark Pool, and Dedicated SQL Pool.
- Strong experience with Azure Data Lake Storage Gen2, Delta Lake on Azure, and Synapse Lakehouse patterns.
- Hands-on experience with Azure Databricks, Delta Live Tables, Delta Lake, PySpark, and Spark SQL.
- Experience migrating workloads from legacy data warehouses or Synapse environments to a Databricks Lakehouse.
- Experience with Unity Catalog for data governance, lineage, and control frameworks.
- Experience with Oracle GoldenGate Replication for real-time source integration.
- Experience with Azure Analysis Services and Power BI consumption-layer patterns.
- Deep understanding of medallion architecture, including Raw, Harmonized, Conformed, and Consumption layers.
- Strong knowledge of SCD Type 0/1/2, CDC patterns, soft/hard deletes, retroactive change processing, deduplication, business reconciliation, and SCD versioning.
- Experience with Synapse SQL Pool, including stored procedures, control tables, and data quality validation patterns.
- Experience with audit, balance, reconciliation, audit trails, data quality controls, and parameterized modular pipeline governance frameworks at enterprise scale.
- Ability to re-implement governance and control frameworks natively in Databricks, including audit logging, reconciliation, and data quality checks.
- Experience with Delta Lake features such as MERGE, CDC, time travel, schema enforcement, and schema evolution.
- Strong Python, SQL, PySpark, Spark SQL, ETL/ELT, and data modeling skills.
- Experience with ETL/ELT at scale, including denormalization, surrogate keys, directory tables, and curated data models.
- Experience integrating complex data sources such as Oracle DB, SQL Server, Azure SQL DB, file systems, Salesforce, and APIs.
- Familiarity with config-driven and automation-first pipeline patterns using YAML, PySpark, SQL-driven generation, and mapping documents.
- Experience with CI/CD pipelines for data engineering using Azure DevOps or GitHub Actions.
- Experience with Infrastructure as Code, such as Terraform or ARM.
- Exposure to containerization tools such as Docker.
- Experience with automated testing frameworks for data pipelines, including unit testing and reconciliation-based validation.
- Comfortable operating across both hands-on engineering and technical architecture.
- Strong communication skills with the ability to engage business, governance, architecture, and engineering stakeholders with clarity.
Preferred Skills
- Experience with Azure Purview for data cataloging and enterprise data governance.
- Exposure to real-time and streaming pipelines such as Event Hub, Kafka, or Kinesis.
- Experience with GenAI or ML platform integration, including MLOps and feature engineering pipelines.
- Familiarity with monitoring and observability tools such as Dynatrace.
- Exposure to BI tools such as Power BI and Tableau.
- Experience working in financial services or other regulated enterprise-scale environments is desirable.
Additional Preferences
- Immediate joiners or candidates serving a short notice period will be preferred.
- Candidates willing to work from Thiruvananthapuram or Pune in hybrid mode will be preferred.
Why Join Claidroid
This is an excellent opportunity to work on enterprise-scale data platform transformation initiatives and grow your career with a globally expanding organization.
