Job Description
Job Purpose
The New York Stock Exchange, part of Intercontinental Exchange (ICE), is seeking a Lead Developer with strong expertise in AWS cloud engineering, Airflow-based orchestration, and modern DevOps practices. In this role, you will lead design, development, automation, and optimization of cloud‑native data and compute pipelines that support high‑availability enterprise workloads.
This is a hands‑on leadership role with significant architectural influence, ownership of cloud platform development, and the opportunity to mentor engineers while shaping how the organization builds and operates large‑scale systems.
Responsibilities
- Cloud Engineering & Architecture (Primary Focus)
- Lead end‑to‑end design and implementation of AWS cloud solutions including compute, networking, storage, IAM, monitoring, and automation.
- Architect highly available, scalable, and secure environments using services such as: EC2, ECS/EKS, Lambda, S3, RDS/Aurora, DynamoDB, CloudWatch, CloudTrail, IAM, Step Functions, Secrets Manager, KMS.
- Define IaC standards and build reusable automation using Terraform or CloudFormation.
- Drive cloud modernization initiatives including containerization, serverless adoption, and event-driven architecture.
- Own environment strategy across dev/QA/prod, including VPC design, cost optimization, and infrastructure reliability.
- Airflow Development & DevOps (Primary Focus)
- Lead the development and operation of Apache Airflow orchestration workflows for data and application pipelines.
- Design DAGs that meet enterprise‑grade reliability, dependency management, observability, and SLA requirements.
- Build CI/CD pipelines to automate Airflow deployments, testing, and versioning.
- Optimize Airflow performance, including scheduler tuning, worker scaling, plugin management, and DAG run efficiency.
- Integrate Airflow with AWS services like S3, EMR, Glue, Lambda, Redshift, and on‑prem systems when required.
- Leadership & Cross‑Functional Collaboration
- Provide technical leadership and mentoring to engineering teams across cloud, data, and application development.
- Conduct design reviews, code reviews, and guide best practices for cloud architecture and DevOps engineering.
- Work closely with operations teams to ensure resilient, monitored, and supportable systems.
- Collaborate with product, business, and security teams to translate requirements into scalable cloud designs.
- Drive engineering excellence through automation, observability, documentation, and process improvements.
Knowledge And Experience
- Bachelor's degree in Computer Science, Engineering, or related field.
- 8+ years of software engineering experience, with 3+ years in a technical leadership or Lead Developer role.
- Deep expertise in AWS cloud architecture and hands‑on experience deploying production workloads.
- Strong, hands‑on experience in Airflow (designing DAGs, writing operators/hooks, administration, scaling).
- Proficiency in Python for backend automation, Airflow development, and cloud tooling.
- Strong knowledge of CI/CD pipelines, Git-based workflows, and DevOps practices.
- Experience with Docker, container orchestration (ECS/EKS), and microservices.
- Strong understanding of secure cloud design, IAM principles, and compliance practices.
Preferred Knowledge And Experience
- Experience with data engineering stack (Glue, EMR, Kafka, Kinesis, Redshift).
- Infrastructure automation using Terraform preferred.
- Experience managing hybrid cloud/on‑prem connectivity.
- Strong monitoring/observability experience (CloudWatch, Prometheus, Grafana, ELK).
- Background in regulated industries or mission‑critical systems.
