Data Engineer

AWS Data Stack, APIs & Agentic AI

Location : Bangalore

Experience : 7 years above

Level : SAL2

Key Responsibilities

Build and maintain batch and incremental pipelines in Python/PySpark on AWS Glue. You'll own their freshness, completeness and cost.
Fetch data from APIs: build reliable ingestion from internal and third-party REST and GraphQL APIs. Handle authentication (API keys, OAuth 2.0), pagination, rate limits, retries with backoff, incremental extraction and schema changes.
Integrate APIs: connect the data platform with source systems, SaaS platforms and partner endpoints using APIs, webhooks and event-driven flows (EventBridge, SQS, Lambda).
Build APIs: design, build and deploy data APIs that serve curated datasets to applications and other teams, using Amazon API Gateway, Lambda and FastAPI. Own versioning, authentication, documentation (OpenAPI) and monitoring.
Bring data from internal systems, APIs and files into the S3 data lake. Manage partitioning, schema changes and Apache Iceberg tables.
Turn raw data into curated, documented layers (raw → staging → mart) that analysts can query directly in Athena.
Use SageMaker Unified Studio to develop, catalogue and publish data assets: projects, SageMaker Lakehouse, SageMaker Catalog, notebooks and query editors.
Make governed data available to AI agents. Build MCP servers or tool endpoints on top of curated datasets, and support Amazon Bedrock agents, knowledge bases and RAG use cases.
Use AI coding agents (Amazon Q Developer, Claude, Kiro, Cursor or similar) to develop, refactor, test and document code. Code review and testing are still required before anything goes live.
Orchestrate workflows with Airflow (MWAA), Step Functions or Glue Workflows, including retries, backfills and alerting.
Set up data quality checks for schema, freshness, row counts and reconciliation with source systems. Trace issues to their root cause.
Apply security and governance: least-privilege IAM, Lake Formation / SageMaker Catalog permissions, encryption, secrets management for API credentials, PII handling and retention.
Tune Athena and Glue for performance and cost through partition design, file compaction, columnar formats and workgroup limits.
Keep all code in Git with CI/CD, and deploy infrastructure as code.

Required Qualifications

Bachelor's degree in Computer Science, Engineering or a related field.
3+ years of hands-on data engineering on production pipelines.
Python (strong): production-quality code, pytest, PySpark, pandas, boto3.
SQL (strong): window functions, CTEs, incremental logic, tuning queries on large partitioned tables.
APIs:
Building APIs: has designed and built REST APIs with FastAPI or Flask, and deployed them on AWS (API Gateway + Lambda, or containers)
Fetching data from APIs: has built production ingestion from REST/GraphQL APIs with requests/httpx. Handles pagination, rate limits, retries and incremental loads
API integration: OAuth 2.0 and API-key authentication, JSON/XML payloads, webhooks, and OpenAPI/Swagger specs
AWS data stack:
AWS Glue: ETL jobs, crawlers, Data Catalog, Glue Studio
Amazon Athena: query authoring, workgroups, federated queries, cost controls
Amazon SageMaker Unified Studio: projects, Lakehouse, Catalog, unified notebooks and query editor
Amazon S3: data lake layout, partitioning, lifecycle policies
Supporting services: Lambda, API Gateway, Step Functions, EventBridge, IAM, CloudWatch, Secrets Manager, Lake Formation
Agentic and AI tools: has used AI coding agents in real work, and understands Amazon Bedrock (agents, knowledge bases), Model Context Protocol (MCP), tool and prompt design, and RAG.
Workflow orchestration with Airflow, Step Functions or an equivalent tool.
Data modelling (dimensional and event-based), plus Git and CI/CD practice.

Preferred Qualifications

Experience running Apache Iceberg in production (compaction, time travel).
Streaming with Kinesis, MSK (Kafka) or Spark Structured Streaming.
Amazon AppFlow or managed connectors for SaaS ingestion (e.g. Salesforce, Google Analytics).
GraphQL API development; async Python (asyncio, aiohttp) for high-volume API extraction.
Redshift / Redshift Serverless or EMR.
dbt; Terraform or CDK.
Has built AI agents with Bedrock AgentCore, Strands Agents SDK, LangGraph or custom MCP servers.
Vector stores and embedding pipelines (OpenSearch Serverless, pgvector).
AWS certification: Data Engineer Associate or Machine Learning Engineer Associate.
Experience with a BI tool such as QuickSight or Power BI.

Competencies

Delivery focus: ships working pipelines and sees operational issues through to resolution.
Rigour: checks numbers before publishing them.
Judgment with AI tools: uses agents to move faster, but reviews, tests and owns everything that goes to production.
Curiosity: keeps up with the fast-changing AWS data and AI stack.
Collaboration: communicates clearly with analysts and the business, and documents work so others can reuse it.

Data Engineer AWS Data Stack, APIs & Agentic AI