What are we looking for We are seeking an experienced and highly skilled Airflow Orchestration Specialist to play a pivotal role in our enterprise data modernization programs. This role will focus on migrating legacy workflows from Oozie to Airflow, orchestrating data pipelines on AWS, and optimizing data ingestion and transformation processes leveraging Databricks.
Key Responsibilities:
Workflow Migration:
- Analyze and convert Oozie workflows into Airflow DAGs using Python-based orchestration.
- Design and implement reusable, modular, and optimized Airflow pipelines for data ingestion, transformation, and orchestration.
- Maintain a one-to-one mapping between legacy workflows and Airflow DAGs, ensuring no data loss or business interruption.
Cloud Data Migration:
- Collaborate with data engineering teams to migrate Cloudera Hadoop workloads to Databricks on AWS.
- Leverage Airflow for scheduling and orchestrating data workflows on AWS-based services (e.g., S3, EMR, Glue, Redshift).
Pipeline Optimization:
- Optimize data ingestion pipelines to achieve high throughput and low latency on AWS cloud infrastructure.
- Integrate Airflow workflows with Databricks for data transformations and analytics.
Error Handling and Monitoring:
- Implement robust error-handling mechanisms, task retries, and alerting within Airflow workflows.
- Set up monitoring dashboards using tools like CloudWatch, Prometheus, or Airflow's built-in features.
Collaboration and Documentation:
- Work closely with data architects, cloud engineers, and DevOps teams to align on migration goals and architecture.
- Document the migration process, workflow logic, and best practices for ongoing maintenance.
Performance Testing:
- Conduct performance testing and benchmarking of migrated pipelines to ensure efficient resource utilization on Databricks and AWS.
CI/CD Implementation:
- Design and maintain CI/CD pipelines for orchestrated workflows using Jenkins and Terraform.
- Automate deployment of Airflow DAGs and infrastructure components using Terraform IaC (Infrastructure as Code).
- Implement quality checks and validation for workflow pipelines during the deployment process.
Technical Expertise:
- Strong experience in Apache Airflow, including designing and managing complex DAGs.
- Hands-on experience with Apache Oozie and migration to modern orchestration tools.
- Proficiency in Python for writing Airflow DAGs and custom operators/hooks.
- Experience with Hadoop ecosystems (Cloudera distribution preferred) and their components like HDFS, Hive, and Spark.
- Solid experience with CI/CD tools, including Jenkins for pipeline automation and Terraform for data orchestration/pipeline provisioning.
- Familiarity with Databricks (on AWS preferred) and its integration with Airflow for ETL and data processing.
Cloud and Infrastructure:
- Solid understanding of AWS services such as S3, EMR, Glue, Lambda, Redshift, and IAM.
- Experience with containerization tools (Docker, Kubernetes) and CI/CD pipelines for workflow deployment.
Analytical and Problem-Solving:
- Ability to debug and resolve issues in data workflows and orchestrators.
- Experience optimizing workflows for performance and scalability.
Preferred Qualifications:
- Experience in large-scale cloud migrations, specifically from on-premises Hadoop to Databricks on AWS.
- Knowledge of Spark and PySpark for big data transformations.
- Familiarity with version control tools (e.g., Git) and workflow monitoring tools.
- Certifications in AWS (e.g., AWS Certified Solutions Architect) or Databricks.
