We are seeking an experienced CI/CD Engineer or Lead to design, automate, and govern delivery pipelines across the Azure ecosystem (Azure DevOps, Databricks, ADF/Synapse) and adjacent CI/CD platforms. While the final tool choice may vary, the candidate must have significant hands-on experience with Jenkins and strong DevOps fundamentals. The role includes in-sprint testing enablement, platform governance (monitoring, administration, configuration), and cost-effective technology selection . Excelle nt communication and documentation skills are essential.
Key Responsibilities
- Pipeline Architecture Delivery: Design and maintain end-to-end CI/CD pipelines using Azure DevOps, Jenkins, GitHub Actions, or GitLab CI. Implement linting and code review checks, integrate static analysis tools (SonarQube/ CodeQL ), architect multi-stage workflows, build reusable templates for Databricks, and manage artifact repositories.
- Azure Platform Governance Operations: Monitor, administer, and configure Azure services (Key Vault, Storage/ADLS, ADF/Synapse, Databricks, AKS). Apply RBAC, secrets management, observability (Azure Monitor, Log Analytics), and cost governance.
- Source Control ALM: Define branching strategy ( GitFlow /Trunk), enforce PR policies in Bitbucket, and integrate Jira workflows with CI/CD.
- Automated In-Sprint Testing: Embed unit, integration, SIT, and E2E/regression tests into pipelines for in-sprint execution. Collaborate with QA to align automated test coverage with sprint goals.
- DevSecOps Policy as Code: Integrate SAST/DAST, dependency scanning, SBOM generation, and enforce quality/security gates. Apply policy-as-code (OPA/ Conftest , Azure Policy).
- Infrastructure as Code ( IaC ): Provision and promote infrastructure using Terraform (preferred) and/or Bicep; manage remote state and modules. Standardize container builds and AKS deployments.
- Data Storage Layer Management Governance: Govern data storage across Unity Catalog (Databricks), Azure SQL / SQL Server, and ADLS. Implement data lineage, optimize storage tiers, and coordinate schema migrations.
- Communication Collaboration: Communicate effectively with cross-functional teams including QA, Development, Product Management, and Architecture. Produce clear documentation, runbooks, and present status updates to stakeholders.
- Leadership (for Lead level): Define CI/CD standards, mentor engineers, lead design reviews, and drive continuous improvement.
Required Qualifications
- Significant hands-on experience with Jenkins (declarative pipelines, shared libraries).
- Hands-on with Azure services (Key Vault, Storage, ADF, Databricks, AKS).
- Expertise in CI/CD best practices across multiple tools (Azure DevOps, GitHub Actions, GitLab CI).
- Solid understanding of in-sprint testing and test automation frameworks.
- Experience with linting, static analysis, and enforcing code quality gates.
- Scripting: Python, Bash, YAML proficiency .
- Familiarity with IaC using Terraform (preferred) and containerization (Docker/Kubernetes).
- Strong communication and documentation skills .
Preferred Qualifications
- Databricks asset bundling/ dbx , MLflow model promotion; Unity Catalog administration.
- Data quality frameworks (e.g., Great Expectations).
- GitOps (Argo CD/Flux), Helm, Kustomize .
- SRE practices; DORA metrics instrumentation.
Cost-effective Tools Technologies
- CI/CD: GitHub Actions, GitLab CI, Jenkins (self-hosted), CircleCI .
- Code Quality: SonarQube/ CodeQL , pre-commit, Black/Flake8, ESLint .
- Observability: Azure Monitor, Log Analytics; Cost Management tools like Azure Advisor.
- IaC : Terraform (preferred), Bicep; Secrets via Azure Key Vault.
- Data Storage: Unity Catalog, Azure SQL / SQL Server, ADLS Gen2; Catalog alternatives like DataHub or Azure Purview.
KPIs / Success Metrics
- Lead time for changes, deployment frequency, change failure rate, MTTR (DORA).
- % of pipelines with linting, static analysis, test/security gates.
- % infra under Terraform; environment drift incidents; secrets coverage.
- Platform cost adherence: budget vs. actual; ingestion/storage optimization achieved.
- Data governance adoption: catalog completeness, Unity Catalog policies applied.
This job is provided by Shine.com
