Key Responsibilities
- Data Solution Design and Development :
- Design and develop scalable and high-performance data pipelines using Azure Data Factory (ADF).
- Implement data transformations and processing using Azure Databricks.
- Develop and maintain NoSQL data models and queries in Cosmos DB.
- Optimize data pipelines for performance, scalability, and cost efficiency.
- Data Integration and Architecture:
- Integrate structured and unstructured data from diverse data sources.
- Collaborate with data architects to design end-to-end data flows and system integrations.
- Implement data security, governance, and compliance standards.
- Performance Tuning and Optimization:
- Monitor and tune data pipelines and processing jobs for performance and cost efficiency.
- Optimize data storage and retrieval strategies for Azure SQL and Cosmos DB.
- Collaboration and Mentoring:
- Collaborate with cross-functional teams including data testers, architects, and business analysts.
- Conduct code reviews and provide constructive feedback to improve code quality.
- Mentor junior developers, fostering best practices in data engineering and cloud development.
Primary Skills
- Data Engineering: Azure Data Factory (ADF), Azure Databricks.
- Cloud Platform: Microsoft Azure (Data Lake Storage, Cosmos DB).
- Data Modeling: NoSQL data modeling, Data warehousing concepts.
- Performance Optimization: Data pipeline performance tuning and cost optimization.
- Programming Languages: Python, SQL, PySpark
Secondary Skills
- DevOps and CI/CD: Azure DevOps, CI/CD pipeline design and automation.
- Security and Compliance: Implementing data security and governance standards.
- Agile Methodologies: Experience in Agile/Scrum environments.
- Leadership and Mentoring: Strong communication and coaching skills for team collaboration.
Soft Skills
- Strong problem-solving abilities and attention to detail.
- Excellent communication skills, both verbal and written.
- Effective time management and organizational capabilities.
- Ability to work independently and within a collaborative team environment.
- Strong interpersonal skills to engage with cross-functional teams.
Educational Qualifications
- Bachelors degree in Computer Science, Engineering, Information Technology, or a related field.
- Relevant certifications in Azure and Data Engineering, such as:
- Microsoft Certified: Azure Data Engineer Associate
- Microsoft Certified: Azure Solutions Architect Expert
- Databricks Certified Data Engineer Associate or Professional
