Data Engineer

Job Title - Data Engineer

Years of Experience - 5 to 8 Years

Location - Bangalore

Notice Period - Immediate to 30 Days

About the Role

We are looking for a skilled Data Engineer with hands-on experience in building and managing scalable data pipelines across cloud platforms. The ideal candidate should have strong expertise in ETL frameworks and exposure to multiple cloud environments.

Key Responsibilities

• Design, develop, and maintain scalable end-to-end ETL/ELT data pipelines (batch and near real-time) across AWS, Azure, and GCP environments (AWS preferred).

• Implement robust data ingestion patterns including CDC, API-based, file-based, and streaming integrations ensuring data quality, reliability, and performance.

• Develop and maintain cloud-native data solutions using MySQL/SQL Server, Azure Data Factory, MFTS, GCP BigQuery, and GCS (preferred: AWS Glue, Redshift, EC2, RDS).

• Build and manage workflow orchestration using Apache Airflow (or equivalent tools) for scheduling, monitoring, and dependency management.

• Work on data ingestion, transformation, and integration across multiple enterprise systems and platforms.

• Develop CI/CD pipelines for data infrastructure and ETL code using Git-based workflows and Infrastructure-as-Code (IaC) best practices.

• Collaborate with cross-functional teams to understand data requirements and deliver scalable and efficient data solutions.

• Work closely with security and compliance teams to implement encryption, IAM least privilege access, network segmentation (VPN/VPC), and governance controls.

• Build monitoring, alerting, and auditing mechanisms for data pipelines including latency tracking, failure monitoring, retries, DLQ, and data quality metrics.

• Optimize data storage, partitioning, and query performance for analytics workloads in BigQuery and Redshift environments.

• Troubleshoot production issues, perform root-cause analysis, and implement corrective and preventive actions.

• Produce and maintain technical documentation including architecture diagrams, runbooks, and data contracts while supporting onboarding and mentoring activities.

• Provide ongoing sustainment and operational support for production data platforms and pipelines.

Required Qualifications

• 5–8 years of total experience in data engineering with at least 3–5 years specifically building cloud-native data pipelines

• Strong experience with Azure Cloud services, especially Azure Data Factory (ADF), data ingestion, and transformation workflows

• Hands-on expertise with GCP services such as BigQuery, Google Cloud Storage (GCS), and cloud-native analytics solutions

• Proficiency in Apache Airflow for workflow orchestration, scheduling, monitoring, and dependency management

• Experience implementing batch and near real-time data pipelines using CDC, API, file based, and streaming ingestion patterns

• Strong understanding of data engineering concepts, data modeling, pipeline optimization, and performance tuning

• Experience with CI/CD pipelines, Git-based workflows, and Infrastructure-as-Code (IaC) practices

• Proficiency in SQL and scripting languages such as Python for data processing and automation

• Experience with monitoring, troubleshooting, and production support for enterprise data platforms 

Preferred Qualifications

• Exposure to AWS services such as Glue, Redshift, EC2, and RDS

• Experience working in multi-cloud enterprise environments (Azure, GCP, AWS)

• Knowledge of data governance, IAM, encryption, and security best practices

• Experience handling large-scale distributed and analytics data systems

• Strong problem-solving skills and experience working in Agile environments