Azure Data Engineer

Azure Data Sources: Azure Data Lake Storage (ADLS), Blob Storage, Azure SQL Database, Synapse Analytics. External Sources: APIs, on-prem databases, flat files (CSV, Parquet, JSON).

Tools: Azure Data Factory (ADF) for orchestration, Databricks connectors.

Apache Spark: Strong knowledge of Spark (PySpark, Spark SQL) for distributed processing.

Data Cleaning & Normalization: Handling nulls, duplicates, schema evolution.

Performance Optimization: Partitioning, caching, broadcast joins.

Delta Lake: Implementing ACID transactions, time travel, and schema enforcement.

Azure Data Factory (ADF): Building pipelines to orchestrate Databricks notebooks.

Azure Key Vault: Secure credential management.

Azure Monitor & Logging: For ETL job monitoring and alerting.

Networking & Security: VNET integration, private endpoints