Job description
- Develop and manageETL workflowsusingAzure Data Factory (ADF).
- Design and implementdata pipelinesusingPySpark on Azure Databricks.
- Work withAzure Synapse Analytics, Azure Data Lake, and Azure Blob Storagefor data ingestion and transformation.
- OptimizeSpark jobsfor performance and scalability inDatabricks.
- Automatedata workflowsand implementerror handling & monitoringin ADF.
- Collaborate withdata engineers, analysts, and business teamsto understand data requirements.
- Implementdata governance, security, and compliancebest practices in Azure.
- Debug and troubleshootPySpark scripts and ADF pipeline failures.
- 4+ yearsof experience inETL developmentwithAzure Data Factory (ADF).
- Hands-on experience withAzure DatabricksandPySparkforbig data processing.
- Strong knowledge ofAzure services
- Proficiency inPythonandPySparkfor data transformation and processing.
- Experience withCI/CD pipelinesfor deploying ADF pipelines and Databricks notebooks.
- Strong expertise inSQLfor data extraction and transformations.
- Knowledge ofperformance tuning in Sparkandcost optimization on Azure.
Skills
Azure Data Factory,Pyspark,Azure
