Lead Data Engineer

Responsibilities

• Active contributor to cost optimization of platforms and services.
• Manage and scale Azure Data Platforms to support new product launches and drive Platform Stability and Observability across data products.
• Build and own the automation and monitoring frameworks that captures metrics and operational KPIs for Data Platforms for cost and performance.
• Responsible for implementing best practices around systems integration, security, performance and Platform management.
• Empower the business by creating value through the increased adoption of data, data science and business intelligence landscape.
• Collaborate with internal clients (data science and product teams) to drive solutioning and POC discussions.
• Evolve the architectural capabilities and maturity of the data platform by engaging with enterprise architects and strategic internal and external partners.
• Develop and optimize procedures to production Alize data science models.
• Define and manage SLAs for Platforms and processes running in production.
• Support large-scale experimentation done by data scientists.
• Prototype new approaches and build solutions at scale.
• Research in state-of-the-art methodologies.
• Create documentation for learnings and knowledge transfer.
• Create and audit reusable packages or libraries.

Qualifications

• 8+ years of overall technology experience that includes at least 4+ years of hands-on software development, Program management, and data engineering
• 4+ years of experience with Data Lake Infrastructure, Data Warehousing, and Data Analytics tools.
• 4+ years of experience in Databricks optimization and performance tuning
• Experience in managing multiple teams and coordinating with different stakeholders to implement the vision of the team.
• Fluent with Azure cloud services. Azure Certification is a plus.
• Experience with integration of multi cloud services with on-premises technologies.
• Experience with data modeling, data warehousing, and building high-volume ETL/ELT pipelines.
• Experience building/operating highly available, distributed systems of data extraction, ingestion, and processing of large data sets.
• Experience with at least one MPP database technology such as Redshift, Synapse or SnowFlake.
• Experience with version control systems like Github and deployment & CI tools.
• Experience with Azure Data Factory, Azure Databricks.
• Experience with Statistical/ML techniques is a plus.
• Experience with building solutions in the retail or in the supply chain space is a plus
• Understanding of metadata management, data lineage, and data glossaries is a plus.
• Working knowledge of agile development, including DevOps and DataOps concepts.
• Familiarity with business intelligence tools (such as PowerBI).