Job Summary:
Maimsd Technology is seeking a highly skilled and experienced Lead Data Engineer to design, deploy, and maintain our entire data infrastructure, data products, and data pipelines. This role requires leveraging software engineering principles to create fully automated, resilient, modular, flexible, scalable, reusable, and cost-effective data transformation pipelines. The Lead Data Engineer will oversee a variety of storage and computation technologies within the Microsoft Azure ecosystem, handling diverse data types and volumes.
Key Responsibilities:
- Oversee the entire data infrastructure to ensure scalability, operational efficiency, and resiliency.
- Mentor junior data engineers within the organization, fostering their growth and skill development.
- Design, develop, and maintain robust data pipelines and ETL processes using a comprehensive suite of Microsoft Azure services, including but not limited to Azure Data Factory, Azure Synapse, Azure Databricks, and Azure Fabric.
- Effectively utilize Azure data storage accounts (e.g., Azure Data Lake Storage Gen 2 & Azure Blob storage) for organizing and maintaining data pipeline outputs.
- Collaborate extensively with data scientists, data analysts, data architects, and other stakeholders to deeply understand data requirements and deliver high-quality, impactful data solutions.
- Optimize data pipelines within the Azure environment for maximum performance, scalability, and reliability.
- Ensure stringent data quality and integrity through the implementation of advanced data validation techniques and frameworks.
- Develop and maintain thorough documentation for all data processes, configurations, and best practices.
- Proactively monitor and troubleshoot data pipeline issues to ensure timely resolution and minimize downtime.
- Stay current with industry trends and emerging technologies to ensure our data solutions remain cutting-edge and competitive.
- Manage the CI/CD (Continuous Integration/Continuous Deployment) process for deploying and maintaining data solutions.
Background & Skills:
- Proven experience in a senior or lead role, utilizing software engineering principles for data infrastructure.
- Strong expertise in designing, developing, and maintaining fully automated data transformation pipelines.
- In-depth practical experience with a wide variety of storage and computation technologies.
- Extensive hands-on experience with Microsoft Azure data services such as Azure Data Factory, Azure Synapse, Azure Databricks, and Azure Fabric.
- Proficient in using Azure data storage solutions like Azure Data Lake Storage Gen 2 and Azure Blob storage.
- Demonstrated ability to optimize data pipelines for performance, scalability, and reliability in a cloud environment.
- Experience with data quality and integrity assurance through validation techniques.
- Familiarity with CI/CD processes for data solutions.
- Excellent collaboration and communication skills for working with cross-functional teams (data scientists, analysts, architects).
- Strong problem-solving abilities and a commitment to continuous learning in emerging data technologies.
