Search by job, company or skills

I

Senior Data Engineer

insurance information bureau of india
Hyderabad, India
5-7 Years
Save
  • Posted 4 days ago
  • Be among the first 10 applicants
Early Applicant

Job Description

Location: Hyderabad, India

Reporting Manager: Head of Analytics

Work Pattern: Full Time, 5 days in the office

Minimum Experience as a data engineer: 5 years

The senior data engineer is a data professional who uses her/his expertise in data engineering and programming to build systems that collect, manage and convert raw data into usable information for data analysts, data scientists and business analysts. Construct datasets that are easy to analyse and support company requirements. Combine raw information from different sources to create consistent and machine-readable formats. Develop and test architectures that enable data extraction and transformation for predictive or prescriptive modelling.

Key Responsibilities

Lakehouse & Medallion Architecture

  • Design and implement Medallion architecture (Bronze, Silver, Gold layers) for scalable data processing with focus on schema enforcement, audit trails, versioning, time travel and incremental processing strategies.
  • Design and maintain optimized data models for Lakehouse and warehouse consumption

Spark Engineering & Performance Optimization

  • Develop high-performance distributed data pipelines using Apache Spark
  • Optimize Spark workloads through partitioning, caching, broadcast joins, and query tuning, execution plans analysis and memory optimization
  • Implement efficient incremental data processing and change data capture strategies
  • Monitor and troubleshoot Spark job failures, latency, and resource bottlenecks

CI/CD & Data Pipeline Automation

  • Design and implement CI/CD pipelines for data engineering workflows. Automate build, test, and deployment of data pipelines across environments
  • Enable infrastructure-as-code and reproducible deployments

Data Pipeline Engineering

  • Build scalable batch and streaming pipelines, Implement incremental data loading and watermark-based processing
  • Design resilient ETL/ELT workflows with monitoring and retry mechanisms

Develop reusable data transformation frameworks

  • Implement pipeline observability, logging, and alerting

Technical Skills (Revised)

Core Data Engineering

  • 6+ years building distributed data pipelines
  • Strong Python and SQL expertise
  • Data modeling and data lifecycle management

Spark & Distributed Processing (Mandatory)

  • Strong experience with Apache Spark (PySpark/Scala Spark)
  • Spark performance tuning and optimization
  • Partitioning strategies and file optimization
  • Execution plan analysis and query optimization
  • Distributed computing concepts

Lakehouse & Medallion Architecture

  • Experience implementing Bronze, Silver, Gold data layers
  • Delta Lake / Lakehouse architectures
  • Incremental processing and time travel
  • Data versioning and schema evolution

CI/CD & DevOps for Data

  • Git-based version control workflows
  • CI/CD tools (Azure DevOps / GitHub Actions / Jenkins or similar)
  • Deployment automation and environment management
  • Infrastructure-as-code concepts

Data Platform & Cloud

  • Experience with Azure/AWS/GCP data platforms
  • Data lakes and analytical warehouses
  • Workflow orchestration tools (Airflow or similar)

Good to Have

  • Microsoft Fabric / Databricks experience
  • Streaming frameworks
  • Data observability tools
  • Insurance or financial domain experience

More Info

Job Type:
Permanent Job
Industry:
Insurance
Function:
Data Engineering
Employment Type:
Full time

Job ID: 150637803

Jobs by Skill - IT
Jobs by Skill - Non IT
International Jobs
Last Updated: 03-07-2026 06:15:45 PM
Homejobs in Hyderabad / Secunderabad, TelanganaSenior Data Engineer

Similar Jobs

Senior Data Engineer I

Alter Domus
Hyderabad, India

Skills:

Power Bi, Apache Spark, Data Governance, Data Modeling, Data Warehousing, Api Integration, Sql, Databricks, Python, Medallion Architecture, ETL processes, Data quality best practices

Senior Data Engineer

Anblicks
Hyderabad, India

Skills:

snowflake , Data Modeling, Databricks, Data Warehousing, Sql, Python, ELT, Etl, Orchestration frameworks, Data pipeline workflow automation, Git-based version control

Senior Data Engineer

Amgen
A
Hyderabad, India

Skills:

Pyspark, Apache Spark, Sql, Apache Airflow, Devops, Gcp, Spark, Databricks, Azure, Python, AWS, data protection regulations, DataOps, Delta Lake, dimensional data modeling, data governance frameworks

Senior Data Engineer - Market #3

NTT Data
India, Remote

Skills:

Github, Sql, Azure Synapse, Apache Spark, Scala

Senior Data Engineer

Cyara
Hyderabad, India

Skills:

Elk, Prometheus, Grafana, Cloudwatch, Docker, Iam, Kubernetes, Python, Aws S3, SCD Type 2 implementation, AWS DMS, Azure Data Migration, AKS, EKS, dimensional modelling