Job Description:
We are looking for a skilled Operations Engineer to support our Azure-based Data Engineering platform powered by Databricks. This role blends DevOps practices, system reliability, and security auditing to ensure our data infrastructure is robust, secure, and scalable. You'll be instrumental in automating workflows, monitoring system health, and enforcing security best practices across our data ecosystem.
Key Responsibilities:
- Maintain and monitor Databricks-based data pipelines and Azure data services (e.g., Data Lake, Synapse, Key Vault).
- Implement CI/CD pipelines using Azure DevOps for data engineering workflows.
- Conduct regular system health checks and performance tuning of clusters and jobs.
- Automate infrastructure provisioning using tools like Terraform or ARM templates.
- Use monitoring tools (e.g., Azure Monitor, Log Analytics, Prometheus) to track system metrics and alert on anomalies.
- Perform security audits using tools like Databricks Security Analysis Tool (SAT) to ensure compliance with best practices.
- Manage secrets and credentials securely using Azure Key Vault and Databricks secret scopes.
- Collaborate with security teams to enforce network security, access control, and data encryption policies.
- Maintain operational runbooks and incident response
Qualifications:
- Bachelor's degree in computer science, Engineering, or related field.
- 5+ years of experience in DevOps, cloud operations, or data platform support.
- Proficiency in Azure services, Databricks, and scripting languages (Python, Bash).
- Experience with CI/CD tools (Azure DevOps, GitHub Actions) and infrastructure-as-code.
- Familiarity with security frameworks and auditing tools in cloud environments.
- Strong troubleshooting skills and a proactive approach to system reliability.
Preferred Skills:
- Experience with Databricks SAT for security posture assessments.
- Knowledge of Unity Catalog, RBAC, and data governance in Azure Databricks.
- Exposure to MLOps and operationalizing machine learning pipelines.
- Understanding of cost optimization strategies in Azure.
Interested candidates should share their updated resume with : [Confidential Information]
