Data Platform Engineering Lead - AITDS

Key Responsibilities

1. Build & Deploy Data Platform Solutions

· Design and develop data pipelines for batch and streaming workloads using Python and Spark/PySpark

· Build and manage AI-ready datasets, feature pipelines, and training data foundations

· Develop backend services and APIs (Python – FastAPI or equivalent) to expose data products

· Implement microservices and event-driven architectures for data ingestion, processing, and serving

· Ensure the platform is scalable, performant, and maintainable

· Mentor engineers on data engineering patterns and best practices.

2. Enable AI/ML & LLM Integration

· Develop AI-ready pipelines for model training, evaluation, and inference

· Enable feature store capabilities and reproducible datasets

· Support integration with LLM/AI services (RAG, embeddings, inference APIs)

· Enable data-to-AI pipelines including vectorization and retrieval workflows.

3. Oversee Implementation on Cloud Infrastructure

· Collaborate with infra teams on Azure, AWS, or GCP

· Build and integrate data lakes, lake-houses, SQL/NoSQL systems

· Enable integration between data platforms and AI/ML systems

· Implement containerized and serverless architectures.

4. Implement Modern Software Engineering Practices

· Implement CI/CD pipelines and observability

· Define and enforce data quality frameworks

· Support metadata, lineage, and governance

· Optimize platform performance, reliability, and scalability.

Required Capabilities / Skills / Experience

· 8+ years in data engineering and backend development

· Strong Python, APIs, and microservices experience

· Deep experience in Spark/PySpark or equivalent

· Expertise in ETL/ELT pipelines and lakehouse architectures

· Experience with feature stores and AI-ready datasets

· Experience with Azure/AWS/GCP

· Familiarity with Docker, Kubernetes, CI/CD

· Familiarity with LLM/GenAI integration patterns

· Familiarity with metadata and governance tooling

· Strong system design and problem-solving skills.