Job Description

Airflow Architect — Infrastructure Automation & Orchestration

Location: Chennai

Experience: 8–10 Years

Environment: On-Premise Data Centre & DR

About the Role

We are looking for an experienced Airflow Architect to design, build and own an enterprise-grade Apache Airflow platform across on-premise Data Centre and DR environments.

The primary focus is infrastructure and operations automation, with data pipelines as a secondary workload.

You will be the technical authority for Airflow, owning architecture, HA/DR, security, performance, upgrades and the engineering team operating the platform.

Qualifications

Bachelor's or Master's degree in Computer Science, Engineering or equivalent.

Key Responsibilities

Architecture & Platform

Design and implement highly available Airflow on-premise.
Architect multi-scheduler, redundant web/API, Celery worker and RabbitMQ infrastructure.
Design and test DC/DR architecture, including RTO/RPO, failover and failback.
Establish Dev/UAT/Pre-Prod/Prod/DR environments.
Own Terraform/Ansible-based Infrastructure as Code and CI/CD for DAG deployment.
Integrate with LDAP/AD/SSO, Vault/CyberArk, PKI and enterprise storage/logging.

Infrastructure Automation

Build reusable Airflow workflows for:

OS/middleware patching and rollback
VM/server/container provisioning
Backup and restore validation
Certificate and credential rotation
Health checks and auto-remediation
Compliance and configuration-drift checks
DR drills and evidence collection
Access provisioning and recertification
Cross-system runbook automation

Develop custom Operators, Hooks, Sensors and Plugins integrating with Ansible, SSH/WinRM, REST APIs, network, storage, backup and monitoring systems.

Implement safe-automation controls such as approval gates, dry-runs, blast-radius limits, maintenance windows and audit trails.

MySQL Metadata Layer

Architect and operate MySQL 8.x for Airflow.
Experience with InnoDB, GTID, replication, InnoDB Cluster/Group Replication and PITR.
Tune connection pools, InnoDB, queries and Airflow metadata tables.
Design backup, restore, failover and schema-migration strategies.

Performance & Reliability

Own Airflow capacity planning and performance tuning.
Optimise schedulers, workers, concurrency, pools and queue performance.
Implement monitoring using Prometheus/StatsD, Grafana, ELK/Loki.
Monitor scheduler health, task latency, queue depth, DB connections and replication lag.

Security & Operations

Own Airflow, Python dependency, OS, MySQL and RabbitMQ vulnerability management.
Handle CVE assessment, Nessus/SonarQube findings and VAPT remediation.
Implement TLS, RBAC, Vault-based secrets, network segmentation and privileged-access controls.
Maintain SOPs, HLD/LLD, runbooks and operational documentation.

Leadership

Lead a team of 8 platform/support engineers.
Conduct architecture and code reviews.
Own production SLAs, incidents, changes and technical roadmap.
Represent the platform in architecture and change-management forums.

Must Have Skills

8–10 years Infrastructure / Platform / DevOps / SRE experience.
5+ years production Apache Airflow experience.
Proven experience building and operating Airflow on-premise, including HA and DR.
Strong MySQL 8.x administration and performance tuning.
Production experience with Celery Executor and RabbitMQ.
Strong Python, Linux and shell scripting.
Hands-on Ansible/AWX and Terraform.
CI/CD experience with Jenkins, GitLab CI or equivalent.
Experience with Grafana, Prometheus/StatsD and ELK/Loki.
Strong understanding of enterprise security, vulnerability management and ITIL processes.
Proven technical leadership and production SLA ownership.