Job Overview
We are seeking an experienced Lead Data Engineer to architect, design, and lead the development of our enterprise data infrastructure. The ideal candidate will have strong technical leadership skills, deep expertise in data engineering technologies, and experience managing data engineering teams across the globe.
Key Responsibilities
Technical Leadership
- Architecture Design: Design and implement scalable, robust data architectures for enterprise-level data processing
- Data Pipeline Leadership: Lead the development of complex ETL/ELT pipelines processing terabytes of data
- Technology Strategy: Evaluate and recommend cutting-edge data technologies and tools
- Performance Optimization: Optimize data processing performance and implement best practices
- Data Quality: Establish and maintain data quality standards and governance frameworks
Team Management
- Team Leadership: Lead and mentor a team of 3-5 Data Engineers
- Code Reviews: Conduct thorough code reviews and establish coding standards
- Technical Mentoring: Guide junior engineers in advanced data engineering concepts
- Project Management: Manage multiple data engineering projects simultaneously
- Stakeholder Communication: Collaborate with product managers, analysts, and business stakeholders
Data Infrastructure
- Cloud Architecture: Design and implement cloud-native data solutions (AWS/Azure/GCP)
- Big Data Processing: Lead implementation of distributed computing frameworks (Spark, Hadoop)
- Real-time Systems: Architect streaming data pipelines using Kafka, Flink, or similar
- Data Warehousing: Design and optimize data warehouses and data lakes
Technical Requirements
Core Skills (Must Have)
- Experience: 5-8 years in data engineering with 2+ team leader role
- Programming: Proficient in level Python, Spark and equivalent programming with experience in data engineering libraries
- SQL Mastery: Advanced SQL skills with experience in PostgreSQL, MySQL, SQL Server, Oracle
- Cloud Platforms: Deep experience with Databricks, Snowflake, AWS, Azure, or Google Cloud Platform
- ETL/ELT: Expert in designing and implementing complex data pipelines
- Big Data: Strong experience with Apache Spark, Hadoop ecosystem
- Version Control: Advanced Git workflows and CI/CD practices
Advanced Skills (Highly Preferred)
- Streaming: Experience with Apache Kafka, Apache Flink, or similar streaming platforms
- Orchestration: Apache Airflow, Prefect, or similar workflow orchestration tools
- Containerization: Docker, Kubernetes for data engineering workloads
- Monitoring: Data pipeline monitoring and observability tools
- Data Modeling: Advanced data modeling techniques (Star Schema, Data Vault, etc.)
- Data Governance: Good understanding on Data Governance and Security implementation on Data Solutions.
Leadership Skills
- Team Management: Proven experience leading data engineering teams
- Technical Communication: Ability to explain complex technical concepts to non-technical stakeholders
- Project Management: Experience managing multiple concurrent projects
- Mentoring: Strong mentoring and coaching abilities
- Problem Solving: Advanced troubleshooting and problem-solving skills
