We are looking for a Lead Data Engineer to join our team and drive the development of data warehouses, data pipelines, ETL processes, and data integrations to support our data analytics and business intelligence needs. In this role, you will design, build, and maintain efficient and reliable data pipelines, transforming raw data into actionable insights. You will work closely with developers, analysts, and business stakeholders to optimize data flow and ensure data accessibility across the organization.
Responsibilities
- Develop, optimize, and maintain scalable data pipelines and ETL processes to support ongoing data analytics, and reporting.
- Build and manage data infrastructure, ensuring the efficient storage, transformation, and retrieval of structured and unstructured data from multiple sources.
- Collaborate with developers, analysts, and other engineers to ensure data accuracy, quality, and availability for analysis and reporting.
- Design, implement, and maintain data architecture, databases, and warehouses, supporting business intelligence and advanced analytics requirements.
- Monitor and troubleshoot data pipeline performance and resolve any issues to ensure data processing efficiency and reliability.
- Develop and enforce data quality standards, data governance policies, and data security protocols across the data lifecycle.
- Leverage cloud platforms (AWS, GCP, Azure) for data engineering tasks, implementing best practices for cloud-native data storage, processing, and pipeline automation.
Requirements
- Strong proficiency in writing and optimizing large, complicated SQL queries, and SQL Server Stored Procedures.
- Min. 4+ years of experience working and building Data-Intensive Applications.
- Proficiency in programming languages such as Python, or Scala for data processing and manipulation.
- Experience with data warehousing solutions like Snowflake, Redshift, or BigQuery.
- Familiarity with cloud platforms (AWS, GCP, or Azure) and their respective data services (e. g., S3 EMR, BigQuery).
- Strong understanding of relational and non-relational databases, including MySQL, PostgreSQL, MongoDB, and Cassandra.
- Experience in SSIS Packages and Power BI reports would be an added advantage.
- Solid understanding of data modeling, data warehousing concepts, and data governance best practices.
- Handling common database requirements such as upgrades, backup, recovery, migration, etc.
- Experience with data pipeline tools such as Apache Airflow, and DBT for ETL orchestration and automation would be an added advantage.
- Familiarity with CI/CD tools and practices for data engineering.
- Experience with data quality frameworks, monitoring, and validation techniques.
This job was posted by Subhanjana Pandey from Indxx.
