We are seeking an experienced Lead Data Engineer to join our dynamic team in Hyderabad. In this role, you will be responsible for the design, development, and maintenance of scalable data pipelines and ETL processes that are critical to supporting our data analytics and machine learning initiatives. You will work with cutting-edge cloud technologies, ensuring data integrity, optimizing infrastructure, and collaborating closely with data scientists and software engineers to deliver robust data solutions.
Responsibilities
- Design, develop, and maintain scalable data pipelines and ETL processes to support data analytics and machine learning initiatives.
- Utilize workflow management platforms to orchestrate and automate data engineering pipelines.
- Implement data models and schemas to support data storage, retrieval, and analysis.
- Work closely with data scientists and analysts to understand data requirements and provide comprehensive data engineering solutions.
- Collaborate with software engineers to integrate data pipelines with existing systems and applications.
- Implement data quality monitoring and validation processes to ensure data integrity and accuracy.
- Optimize data infrastructure and performance to meet scalability and performance requirements.
- Setup instrumentation and monitoring to track production health parameters and proactively alert in case of any performance degradation.
Qualifications
- Over 5 years of experience in data engineering or a related role.
- Strong experience with the Google Cloud Platform (GCP) and its respective data services (BigQuery, Airflow).
- Strong experience working with relational databases such as SQL Server, as well as NoSQL databases such as MongoDB.
- Proficiency in Python for scripting and data manipulation.
- Solid understanding of data modeling concepts and database technologies (e.g., SQL, NoSQL, relational databases).
- Experience working with monitoring tools such as Datadog is a plus.
- Experience with data warehouse solutions such as Snowflake, Redshift, or BigQuery.
- Excellent problem-solving skills and attention to detail.
- Strong communication and collaboration skills, with the ability to work effectively in a team environment.
