Job Description
API Refactoring
Refactor Spark-Scala jobs and libraries currently using the deprecated Log Analytics API.
Integrate with the Azure Monitor Query API for monitoring validation.
Implement the Logs Ingestion API for secure custom data submission.
Databricks & Cluster Monitoring
Ensure Databricks cluster logging, initialization scripts, and custom metrics are integrated with the Azure Monitor Agent (AMA).
Configure and validate Data Collection Rules (DCRs) to replace MMA/OMS-based monitoring.
Custom Metrics & Logging
Implement efficient logging and custom metric collection within Spark-Scala applications.
Package and submit telemetry via Azure Monitor APIs or structured streaming endpoints.
Security Integration
Update authentication and connectivity to use User-assigned Managed Identities (UAMI) for secure access to Azure Monitor services.
Validation & Testing
Develop and execute validation tests within the Databricks environment.
Verify end-to-end data flow to Log Analytics custom tables and ensure acceptable query performance.
Required Skills & Experience
Strong proficiency in Apache Spark development using Scala
Deep hands-on experience with Azure Databricks architecture, cluster configuration, and logging practices
Solid understanding of Azure Monitor, Log Analytics, and Kusto Query Language (KQL)
Experience with RESTful APIs and secure, scalable data integration in Azure
Knowledge of monitoring agents (MMA/OMS and AMA) and configuration using Data Collection Rules (DCRs)
