Roles & Responsibilities:
- Design and develop scalable and efficient data pipelines using Azure Databricks, PySpark, and related technologies
- Work with Azure Integration Services to orchestrate data workflows across various data sources
- Collaborate with data scientists, analysts, and business teams to understand data requirements and deliver solutions
- Develop scripts and automation solutions using Python for data ingestion, transformation, and validation
- Integrate back-end systems with APIs and other data sources to ensure seamless data flow
- Ensure solutions adhere to best practices in performance, security, and compliance
- Leverage RESTful APIs and microservices for seamless data integration
- Participate in code reviews, testing, and debugging to improve data pipeline reliability and performance
- Maintain up-to-date knowledge of emerging cloud and data technologies
