Architect and implement end-to-end data pipelines using Azure Data Factory, Databricks, and Snowflake for large-scale data ingestion, transformation, and storage.
Using Microsoft Azure data PaaS services, design, build, modify, and support data pipelines leveraging DataBricks and PowerBI in a medallion architecture setting.
If necessary, create prototypes to validate proposed ideas and solicit input from stakeholders.
Excellent grasp of and expertise with test-driven development and continuous integration processes.
Analysis and Design Converts high-level design to low-level design and implements it.
Collaborate with Team Leads to define/clarify business requirements, estimate development costs, and finalize work plans.
Run unit and integration tests on all created code Create and run unit and integration tests throughout the development lifecycle.
Benchmark application code proactively to prevent performance and scalability concerns.
Collaborate with the Quality Assurance Team on issue reporting, resolution, and change management.
Support and Troubleshooting Assist the Operations Team with any environmental issues that arise during application deployment in the Development, QA, Staging, and Production environments.
Familiarity with PowerBI and Reltio is advantageous but not required.
Collaborate with BI teams to ensure data models are optimized for reporting in Power BI, with a focus on performance and usability.
Establish data governance, quality, and security controls, ensuring compliance with GDPR, HIPAA, and global clinical data regulations.
Mentor and guide junior engineers, fostering technical excellence and knowledge sharing.
Drive automation and CI/CD practices within data engineering pipelines, integrating with version control and deployment workflows.
Work closely with Data Architects, Business Analysts, and Product Owners to translate business needs into technical solutions.
Skills
Expert-level knowledge of Azure Data Factory, Databricks, and Snowflake.
Understanding of quality processes and estimate methods.
Understanding of design concepts and architectural basics.
Fundamental grasp of the project domain.
The ability to transform functional and nonfunctional needs into system requirements.
The ability to develop and code complicated applications is required.
The ability to create test cases and scenarios based on specifications.
Solid knowledge of SDLC and agile techniques.
Knowledge of current technology and trends.
Logical thinking and problem-solving abilities, as well as the capacity to collaborate.
