- Position: Lead / Senior Data Engineer
- Location: Chennai-Onsite
- Shift: US Eastern Time ( 5:00 PM – 2:00 AM )
- Experience : 8+ years overall
- Interviews: Assessment || 2 Interview rounds.
Notice Period: Immediate Joiner/Serving Notice Period
Job Description
- Job title : Lead / Senior Data Engineer
- Experience: 8+ years
Roles And Responsibilities
Role Overview
- The Lead Data Engineer will be responsible for designing, developing, and delivering high-quality software and data solutions while leading a team of engineers. The role involves hands-on technical work, architectural decision-making, mentoring junior developers, and collaborating with cross-functional teams to ensure successful delivery of scalable data platforms and analytical solutions.
Key Responsibilities
- Lead the end-to-end design, development, and delivery of software systems, data pipelines, and components.
- Define technical strategy, architecture, and best practices for development and data engineering.
- Design and build optimized data pipelines using cutting-edge technologies in a cloud environment.
- Construct infrastructure for efficient ETL processes from various sources and storage systems.
- Architect, design, and maintain database pipeline architectures, ensuring readiness for AI/ML transformations.
- Lead the implementation of algorithms and prototypes to transform raw data into useful information.
- Review code for quality, scalability, and performance.
- Collaborate with Product Managers, Business Managers, Designers, and QA teams to translate business requirements into technical solutions.
- Develop analytical tools, programs, and reporting mechanisms.
- Create data validation methods and data analysis tools.
- Interpret data trends and patterns to establish operational alerts.
- Conduct complex data analysis and present results effectively.
- Prepare data for prescriptive and predictive modeling.
- Ensure compliance with data governance and security policies.
- Troubleshoot, debug, and resolve complex technical issues.
- Drive continuous improvement in software and data development processes, tools, and methodologies.
- Mentor and guide engineers through code reviews, technical discussions, and training.
- Ensure timely delivery of projects while maintaining high engineering standards.
- Continuously explore opportunities to enhance data quality and reliability.
- Apply strong programming and problem-solving skills to develop scalable solutions.
- Demonstrate passion for testing strategy, problem-solving, and continuous learning.
- Willingness to acquire new skills and knowledge.
- Possess a product/engineering mindset to drive impactful data solutions.
- Experience working in distributed environments with global teams.
- Stay current with emerging technologies and industry trends to propose innovative solutions.
Technical Skills And Experience Requirements
- Minimum 8+ years of hands-on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines.
- High proficiency in Python, Scala, and Spark for applied large-scale data processing.
- Expertise with big data technologies, including Spark, Data Lake, Delta Lake, and Hive.
- Solid understanding of batch and streaming data processing techniques.
- Proficient knowledge of the Data Lifecycle Management process, including data collection, access, use, storage, transfer, and deletion.
- Expert-level ability to write complex, optimized SQL queries across extensive data volumes.
- Experience with RDBMS and OLAP databases such as MySQL and Snowflake.
- Familiarity with Agile methodologies.
- Obsession for service observability, instrumentation, monitoring, and alerting.
- Knowledge or experience in architectural best practices for building data lakes.
- Qualifications - Bachelor's degree in computer science, Engineering, Information Systems, or related field.
Skills: python,data lake,delta lake,rdbms,spark,sql,olap,scala,snowflake,data engineer,hive,mysql
