Senior Data Engineer

Location Name: Pune Corporate Office - Mantri

Job Purpose

Build and maintain reliable, scalable batch and real-time data pipelines on the Enterprise Data Platform to enable analytics, reporting, and downstream applications. The role delivers high-quality data engineering solutions using SQL, Python, and PySpark, with strong focus on streaming, Change Data Capture (CDC), and database mirroring to ensure timely, trusted data delivery.

Duties And Responsibilities

• Design, develop, and optimize data pipelines using SQL, Python, and PySpark on cloud data platforms.
• Implement and operate real-time/streaming data ingestion (e.g., Spark Structured Streaming/Kafka) including schema evolution and late-arriving data handling.
• Set up and manage CDC frameworks and database mirroring for nearrealtime replication and minimal-latency updates.
• Build robust data models and curated datasets for analytics, dashboards, and application consumption.
• Ensure data quality, lineage, and observability (validation, alerting, SLAs/SLOs) across batch and streaming workloads.
• Drive performance tuning and cost optimization (partitioning, file formats, caching, autoscaling).
• Harden solutions with security best practices (access controls, PII handling), governance, and compliance standards.
• Contribute to CI/CD using Git/GitHub and DevOps pipelines; automate testing and deployments.
• Partner with Data Platform, BI/Analytics, and Application teams to translate requirements into technical solutions.
• Provide L2/L3 support for pipelines and jobs; troubleshoot incidents, perform RCA, and implement preventive fixes.
• Create and maintain technical documentation and runbooks; participate in code reviews and knowledge sharing.
  
  

Key Decisions / Dimensions

• Select appropriate ingestion patterns (batch vs. streaming), CDC/mirroring approaches, and storage formats.
• Define partitioning, indexing, and optimization strategies to meet SLAs.
• Recommend tooling and frameworks for orchestration, testing, and observability.
• Prioritize defect fixes and enhancements based on impact and risk.
  
  

Major Challenges

• Maintaining reliability and low latency for missioncritical streaming and CDC pipelines.
• Managing schema changes and data drift across diverse source systems.
• Balancing feature delivery with production support within tight timelines.
• Optimizing performance and cost at scale across environments.
  
  

Educational Qualifications


Required Qualifications and Experience

• Graduate or PostGraduate in Computer Science, Information Technology, or Data Science/Technologies.
  
  

Work Experience

• 3–4 years of handson data engineering experience.
  
  

Technical Expertise / Skills Keywords

• SQL, Python, PySpark
• Data streaming (e.g., Spark Structured Streaming, Kafka), CDC (e.g., Debezium/Log-based), Database Mirroring
• Data modeling, performance tuning, and optimization
• Version control (Git/GitHub) and DevOps pipelines (e.g., Azure DevOps)
• Preferred: Azure Databricks, Azure Data Factory, Data Lake Storage; experience with orchestration and observability tools.