Senior AWS Data Engineer (Remote)

Lead Data Engineer, AWS Data Platform

Hiring immediately. We are looking for someone who can start now or on a short notice period.

Experience: 5+ years in data engineering, 1 to 2 of those leading Type: Full-time Location: Remote (India) Working hours: 5:00 PM to 2:00 AM IST. Hard requirement, not a preference.

Stack: AWS (S3, Glue, Redshift, Athena, Lake Formation, SNS/SQS, RDS Postgres), Apache Airflow, a metadata-driven ingestion framework, SQL, Python, GitHub, VS Code, GitHub Copilot

About the role

We run a modern data platform on AWS that feeds analytics and AI/ML teams at an enterprise client. Data moves from a raw landing zone through curated layers into Redshift, driven by a metadata framework rather than hand-written one-off pipelines, and it runs across both non-production and production environments. We want one person accountable for the design, the standards, and the roadmap of that platform. Senior enough to say no to a bad model before it ships, and calm enough to run a backfill without turning it into a war room.

This is an immediate opening on a live project, so we are moving fast on interviews and offers. If you are available on a short notice period, say so in your application.

The evening shift is real and it is the reason this role is hard to fill. Your day overlaps almost entirely with a US-based stakeholder group, and this role talks to them directly rather than through a proxy. If that overlap does not work for you long term, please do not apply. We would rather lose you now than in month three.

What you will own

• The metadata-driven ingestion framework. Extending it, hardening it, and making it the default path for new sources instead of another bespoke pipeline.
• The raw to land to Redshift flow. Layer contracts, partitioning, file formats, compaction, and load patterns that stay reliable as source count grows.
• The Redshift warehouse. Dimensional models, distribution and sort keys, WLM behaviour, and a real answer for cost per query.
• Orchestration in Airflow. DAG standards, retries, SLAs, alerting, dependency design, and backfills that do not become incidents.
• Non-prod and prod parity. Promotion process, environment configuration, and the discipline that keeps the two from drifting apart.
• Governance in Lake Formation. Fine-grained access, row and column controls, and a defensible answer to who can see what and who approved it.
• Event-driven flows on SNS and SQS, including the parts people skip: DLQs, replay, ordering, idempotency.
• RDS Postgres. Schema design, indexing, and query performance for the operational and metadata layers.
• The contract with analytics and ML. Curated datasets and models those teams can build on without messaging you first.
• The engineering bar. Code review, modelling and naming standards, documentation, and mentoring 2 to 4 engineers.

Must have

• 5+ years building production data platforms, including 1 to 2 years as a lead or tech lead accountable for other engineers output, not just your own.
• Hands-on AWS data stack: S3, Glue (jobs, crawlers, catalog), Redshift, Athena, Lake Formation, and RDS Postgres. Ownership, not exposure.
• SNS and SQS used inside real data or event-driven pipelines, including failure handling and replay.
• Experience with metadata-driven or config-driven ingestion frameworks, and a clear view of why they beat one-off pipelines at scale.
• Airflow in production. You have authored DAGs, run backfills, and been the person paged when they failed.
• SQL you can defend line by line, plus genuine dimensional modelling background (Kimball or equivalent).
• Redshift performance work: distribution and sort key decisions, load strategy, and diagnosing a slow query rather than throwing compute at it.
• Comfortable working across separate non-production and production environments with a controlled promotion process.
• Python for pipeline and framework development, with Git-based collaboration and code review.
• Proven remote leadership: async written communication, running reviews, and holding a technical position with stakeholders you have never met in person.
• Ability to work 5:00 PM to 2:00 AM IST consistently.

Nice to have

• Curated data layers or a feature store built for ML teams.
• Infrastructure-as-code for data infra: Terraform, CDK, or CloudFormation.
• Cost governance: Redshift concurrency scaling and WLM tuning, Glue DPU sizing, S3 lifecycle policies.
• dbt or a comparable transformation framework.
• Working fluently with AI coding assistants such as GitHub Copilot.

We review applications daily and respond within 48 hours. If you fit the brief and can start soon, apply now.