Job description
Description: The Senior Data Engineer is responsible for leading the design, development, and optimization of data infrastructure and systems to meet complex business needs. They play a strategic role in developing and managing large-scale data pipelines, ensuring scalability, reliability, and performance. As a key technical expert, the Senior Data Engineer collaborates with cross-functional teams to deliver innovative data solutions and ensures adherence to best practices in data management and governance.
Responsibilities
- Leads the design and development of advanced data pipelines, ensuring seamless ETL/ELT of data into enterprise storage systems.
- Oversees data integration from diverse sources, maintaining data consistency, integrity, and compliance with enterprise standards.
- Transforms raw data into usable formats through cleansing, aggregation, filtering, and enrichment techniques.
- Optimizes data pipelines and workflows for scalability, performance, and cost efficiency.
- Establishes robust data validation, quality checks, and error-handling mechanisms to ensure enterprise data integrity.
- Mentors junior data engineers, promoting technical growth and team development.
- Implements and enforces best practices in data governance, including metadata management and lineage tracking.
- Drives the adoption of real-time data streaming and processing workflows to enable advanced analytics and decision-making.
- Partners with stakeholders to identify and address complex data challenges, delivering scalable and innovative solutions.
- Stays updated with emerging data technologies, evaluating their potential to enhance enterprise capabilities.
- Develops and enforces data security protocols to protect sensitive information.
Job Requirements
Education
- A bachelor s degree in computer science, data science, software engineering, information systems, or related quantitative field; master s degree preferred
Experience
- 8+ years of data engineering experience, including expertise in data integration, pipeline optimization, and enterprise-scale data solutions.
- Proven leadership in implementing big data solutions (e.g., Snowflake, Databricks) and distributed data systems (e.g., Apache Spark, Flink).
Skills
- Advanced proficiency in Apache technologies (Kafka, Airflow, Spark) and programming languages (Python, Java, Scala).
- Expertise in data query tools (SQL, Hive) and database technologies (NoSQL, Hadoop, Teradata).
- Strong knowledge of cloud platforms (AWS, Azure, GCP) and modern data architectures.
- Experience with real-time streaming data workflows and AI/ML analytics initiatives.
- Exceptional analytical and problem-solving skills with debugging expertise in complex systems.
- Excellent communication skills to effectively collaborate with technical and non-technical stakeholders.
- Strong leadership abilities to mentor teams and influence cross-functional initiatives.
