Azure Data Engineer

Azure Data Engineer: 5+ years

Data Management and Storage:

• Design and implement data storage solutions using Azure SQL Database, Azure Data Lake Storage and Azure Blob Storage.
• Ensure data storage systems are scalable, performant, and cost-effective.

Data Integration and ETL:

• Develop and manage ETL (Extract, Transform, Load) processes using Azure Data Factory.
• Ensure efficient data cleaning, transformation, and loading into Data Lakes.

Big Data and Analytics:

• Utilize big data technologies like Azure Databricks and Apache Spark for data processing and analytics.
• Create data processing workflows and pipelines to support data analytics and machine learning applications.

Data Pipeline Architecture:

• Design, construct, and maintain data pipelines and combine large, complex datasets that meet business requirements.

Optimization and Monitoring:

• Implement effective metrics and monitoring processes to ensure data quality and system performance.
• Identify and implement internal process improvements, such as automating manual processes and optimizing data delivery.

Collaboration and Support:

• Work with stakeholders to address data-related technical issues and support data-driven decision-making.
• Collaborate with data and analytics experts to enhance data system functionality.

ML Use Cases:

• Collaborate with data scientists to create datasets for variety of use cases .
• Implement data annotations, curation techniques to enhance data readiness for ML Applications.