Senior Lead Data Engineer
Location: Remote/ Bangalore
Experience: 11+ years
About The Role
Are you passionate about building high-performance data systems Do you thrive at the intersection of Data Engineering, Cloud, and Big Data technologies
We're looking for a
Senior Lead Data Engineer to join our growing team! If you have deep expertise in
ETL/ELT pipelines,
Data Warehousing,
Cloud platforms (GCP, AWS), and
Hadoop ecosystems, we want to hear from you!
ETL Tools: primarily use Pentaho Data Integration (PDI) for designing and managing ETL workflows, including data extraction, transformation, and loading from various sources into target systems.
Databases
- MySQL for transactional data handling and intermediate data staging.
- Amazon Redshift as the primary data warehouse for analytics and reporting workloads.
Workflow Monitoring & Orchestration
- Azkaban is used for monitoring and managing data pipeline executions and ensuring task dependencies are handled correctly.
Job Scheduling & Scripting
- Crontab and Unix shell scripting are utilized for scheduling recurring jobs and automating data processes on a daily or hourly basis.
Cloud Services (AWS)
- Hands-on experience with:
- RDS for managing relational database services.
- Amazon Redshift for handling analytical queries and data warehousing.
- EC2 for running compute workloads.
- Basic IAM (Identity and Access Management) for managing user roles and access permissions.
CI/CD & Deployment
- Jenkins is used for automating the deployment of ETL scripts and scheduling code pushes to production environments.
What You'll Do
- Architect, design, and build robust ETL/ELT pipelines using Informatica PowerCenter and cloud-native solutions.
- Work hands-on with Data Warehousing concepts and Data Modeling to power analytics and insights.
- Build large-scale Hadoop workflows with tools like Hive, Pig, Spark, and HDFS.
- Collaborate closely with business and technical teams to translate requirements into scalable solutions.
- Produce high-quality technical documents: design docs, mapping specs, and unit test plans.
- Optimize big data processing and cloud infrastructure for performance, scalability, and cost.
- Mentor junior engineers and contribute to a culture of technical excellence.
Tech Stack You'll Work With
- Informatica PowerCenter
- GCP (BigQuery, Dataflow, Pub/Sub)
- AWS (Redshift, S3, Glue, Lambda)
- Hadoop (Hive, Pig, Spark, HDFS)
- SQL, Python (bonus!)
- DevOps, CI/CD, Docker/Kubernetes (nice to have!)
What You Bring
- 11+ years of IT experience with a strong focus on Data Engineering.
- Solid foundation in Data Warehouse architecture and Data Modeling principles.
- Deep understanding of Cloud and Big Data technologies.
- Excellent communication, documentation, and troubleshooting skills.
- Ability to thrive in a fast-paced, agile environment.
Your Background
- Bachelor's or Master's in Computer Science, Information Technology, or related field.
