Position Summary:
We are seeking a talented Data Engineer with a strong background in data engineering to join our team. You will play a key role in designing, building, and maintaining data pipelines using a variety of technologies, with a focus on the Microsoft Azure cloud platform.
Responsibilities:
- Design, develop, and implement data pipelines using Azure Data Factory (ADF) or other orchestration tools.
- Write efficient SQL queries to extract, transform, and load (ETL) data from various sources into Databricks.
- Utilize PySpark and Python for complex data processing tasks on large datasets within Azure Databricks.
- Databricks is Mandatory
- Collaborate with data analysts to understand data requirements and ensure data quality.
- Hands-on experience in designing and developing Datalakes and Warehouses
- Implement data governance practices to ensure data security and compliance.
- Monitor and maintain data pipelines for optimal performance and troubleshoot any issues.
- Develop and maintain unit tests for data pipeline code.
- Work collaboratively with other engineers and data professionals in an Agile development environment.
Preferred Skills & Experience:
- Good knowledge of PySpark & working knowledge of Python
- Full stack Azure Data Engineering skills (Azure Data Factory, DataBricks and Synapse Analytics)
- Experience with large dataset handling
- Hands-on experience in designing and developing Datalakes and Warehouses
