Azure Data Engineer: 5+ years
Data Management and Storage:
- Design and implement data storage solutions using Azure SQL Database, Azure Data Lake Storage and Azure Blob Storage.
- Ensure data storage systems are scalable, performant, and cost-effective.
Data Integration and ETL:
- Develop and manage ETL (Extract, Transform, Load) processes using Azure Data Factory.
- Ensure efficient data cleaning, transformation, and loading into Data Lakes.
Big Data and Analytics:
- Utilize big data technologies like Azure Databricks and Apache Spark for data processing and analytics.
- Create data processing workflows and pipelines to support data analytics and machine learning applications.
Data Pipeline Architecture:
- Design, construct, and maintain data pipelines and combine large, complex datasets that meet business requirements.
Optimization and Monitoring:
- Implement effective metrics and monitoring processes to ensure data quality and system performance.
- Identify and implement internal process improvements, such as automating manual processes and optimizing data delivery.
Collaboration and Support:
- Work with stakeholders to address data-related technical issues and support data-driven decision-making.
- Collaborate with data and analytics experts to enhance data system functionality.
ML Use Cases:
- Collaborate with data scientists to create datasets for variety of use cases .
- Implement data annotations, curation techniques to enhance data readiness for ML Applications.
