Search by job, company or skills

HealthEdge

Senior Data Engineer

HealthEdge
Bengaluru, India
5-10 Years
Save
  • Posted an hour ago
  • Be among the first 10 applicants
Early Applicant

Job Description

Overview

 Sr Data Engineer

Position Summary

We are looking for a self-sufficient BI Analyst / Data Engineer to join a small, high-impact analytics team. In this role you will own the full lifecycle of business intelligence delivery—from building and maintaining ETL pipelines in AWS to designing interactive dashboards in Amazon QuickSight. You will work closely with onshore stakeholders to translate business requirements into reliable, well-documented data products that drive operational and strategic decision-making.

The ideal candidate is comfortable operating independently across the data stack, takes ownership of their work, and communicates proactively with team members across time zones.

Core Responsibilities

ETL / Data Pipeline Development

  • Design, build, and maintain AWS Glue (PySpark) ETL jobs that ingest data from multiple source systems such as Salesforce, JIRA, Azure DevOps, and Zendesk into a layered data lake architecture (raw → core → presentation).
  • Develop and optimize AWS Step Functions workflows for pipeline orchestration, including parallel execution, error handling, retry logic, and concurrency management.
  • Implement incremental loading strategies, schema evolution handling, and data quality checks within Glue scripts.
  • Work with Apache Iceberg table format on S3 for ACID-compliant analytics tables, including MERGE operations, compaction, snapshot expiration, and orphan file cleanup.
  • Integrate supporting AWS services such as EventBridge, SQS, SNS, Athena, and CloudWatch into pipeline architectures.
  • Troubleshoot production pipeline failures, diagnose performance bottlenecks (OOM errors, API timeouts, DPU tuning), and implement fixes with minimal guidance.
  • Write and maintain clear technical documentation for all pipeline components.

Dashboard & Reporting Development

  • Build, publish, and maintain Amazon QuickSight dashboards and analyses that serve operational and executive audiences.
  • Create well-modeled QuickSight datasets backed by Athena queries over Iceberg tables, applying appropriate joins, calculated fields, and filters.
  • Translate business requirements and existing Tableau/Excel reports into QuickSight visuals, ensuring accuracy and usability.
  • Optimize dashboard performance, evaluating SPICE vs. Direct Query modes and managing refresh schedules.
  • Collaborate with stakeholders to iterate on dashboard designs, incorporating feedback and refining KPIs.

General Team Responsibilities

  • Participate in code reviews, sprint planning, and knowledge-sharing sessions with the broader BI and engineering teams.
  • Follow established Git branching strategies and CI/CD practices for infrastructure and code deployment.
  • Monitor data freshness and pipeline health; respond to alerts and escalate issues as appropriate.
  • Proactively identify opportunities to improve data quality, reduce pipeline run times, and enhance reporting capabilities.

Required Qualifications

  • 5–10+ years of professional experience in data engineering, BI development, or analytics engineering.
  • Strong hands-on experience with AWS Glue (PySpark), Step Functions, S3, Athena, and IAM.
  • Proficiency in Python and SQL; experience writing PySpark transformations and Athena queries.
  • Experience building and maintaining ETL pipelines that ingest data from REST APIs or SaaS platforms (e.g., Salesforce, JIRA, Zendesk).
  • Working knowledge of Amazon QuickSight, including dataset creation, calculated fields, parameters, and dashboard publishing.
  • Familiarity with columnar/lakehouse table formats such as Apache Iceberg, Delta Lake, or Apache Hudi.
  • Solid understanding of data warehouse concepts: dimensional modeling, slowly changing dimensions, incremental vs. full loads.
  • Comfortable working independently with minimal day-to-day supervision, managing your own task priorities, and communicating status asynchronously.
  • Strong written and verbal English communication skills; able to participate in cross-timezone meetings and write clear documentation.

Preferred Qualifications

  • Experience with infrastructure-as-code tools such as AWS CDK, CloudFormation, or Terraform.
  • Familiarity with CI/CD pipelines (GitHub Actions, CodePipeline) for data engineering workflows.
  • Exposure to PII detection/redaction practices or data governance frameworks.
  • Prior experience translating Tableau workbooks or Excel-based reports into another BI platform.
  • Knowledge of EventBridge-driven architectures and SQS/SNS messaging patterns.
  • AWS certifications (e.g., Data Analytics Specialty, Solutions Architect Associate) are a plus.

What We Offer

  • The opportunity to work on a greenfield BI platform with modern cloud-native tooling.
  • A collaborative, small-team environment where your contributions have direct and visible impact.
  • Exposure to a broad technology stack spanning data engineering, analytics, and cloud infrastructure.
  • Mentorship and growth opportunities within a growing data organization.

More Info

Job Type:
Permanent Job
Industry:
Other
Function:
Data Engineering
Employment Type:
Full time

About Company

HealthEdge

Job ID: 150933379

Jobs by Skill - IT
Jobs by Skill - Non IT
International Jobs
Last Updated: 07-07-2026 08:32:20 PM
Homejobs in Bengaluru / BangaloreSenior Data Engineer

Similar Jobs

Junior & Senior Data Engineer

avp vigilant technology pvt ltd
Bengaluru, India

Skills:

snowflake , Git, Gcp, Apache Kafka, Apache Spark, Azure, Sql, Python, AWS, ELT, Etl

Senior Data Engineer

applied materials india
Bengaluru, India

Skills:

snowflake , Orchestration, Agile Methodology, Pyspark, ELT, RDBMS, Big Data, Scrum, Python, AWS, Cloud Architecture, Redshift, Sql, DevSecOps, Gcp, Spark, Databricks, Azure, Etl, data pipelines, dbt, Kanban, NoSQL databases, unstructured data processing, real-time streaming systems

Senior Data Engineer

Oracle
Bengaluru, India

Skills:

Java, Spark SQL, Data Transformation, Data Modelling, Sql, Azure, Python, AWS, Etl, LLMs, Claude, agent frameworks, prompt engineering, ChatGPT, oci

SENIOR DATA ENGINEER-AZURE AND SNOWFLAKE IMMEDIATE JOINERS

phigital care
Bengaluru, India

Skills:

Data Modeling, Azure Data Bricks, Pyspark, Data Governance, Azure Functions, Terraform, Azure Synapse Analytics, Azure Data Lake, Python, Apache Spark, Api Integration, Sql, Devops, Git, Airflow, Performance tuning and optimization, Key Vault, CI CD, Azure Data Engineering Services, Monitoring and observability frameworks, Snowflake Data Warehouse Development, Logic Apps, Azure Data Factory ADF, DBT Data Build Tool

Senior Data Engineer

themesoft inc.
Bengaluru, India

Skills:

Jenkins, SAP integration, Power Bi, Collibra, Apache Spark, Databricks, Python, Sql, dbt, Delta Lake, Autoloader