Data Scientist

Job Summary

We are looking for two AI Data Engineers to build operate and continuously improve the data pipelines retrieval infrastructure and ML and LLMOps foundations that power our AI initiatives. These professionals will be responsible for turning reference architectures and data contracts into robust production-grade implementations that serve conversational AI assistants dashboard copilots autonomous agents RAG applications and predictive ML models.

Responsibilities

We are looking for two AI Data Engineers to build operate and continuously improve the data pipelines retrieval infrastructure and ML and LLMOps foundations that power our AI initiatives. These professionals will be responsible for turning reference architectures and data contracts into robust production-grade implementations that serve conversational AI assistants dashboard copilots autonomous agents RAG applications and predictive ML models.

The role involves data pipeline engineering including building testing and maintaining production pipelines in batch and real-time environments using Snowflake PySpark Delta Lake and Kafka. Engineers will implement data quality checks schema validation and alerting at every pipeline stage migrate legacy ETL and data warehouse systems to cloud-native AWS or Azure architectures with measurable latency and cost improvements and maintain CI/CD pipelines with automated testing deployment rollback and infrastructure as code using Terraform and GitHub Actions.

They will also work on RAG vector and retrieval infrastructure by building end-to-end retrieval systems covering document ingestion embedding pipelines vector store management with Pinecone FAISS ChromaDB or OpenSearch and hybrid retrieval layers. Responsibilities include implementing chunking metadata filtering and re-ranking tuned for precision recall and latency maintaining data freshness and index consistency and instrumenting with context relevance and faithfulness metrics.

In addition they will support semantic layer and knowledge infrastructure by implementing and maintaining business entity mappings ontologies and knowledge graphs with Neo4j building and versioning feature stores and semantic data contracts serving ML models and LLM applications and managing metadata data lineage and audit trail instrumentation across the platform.

The engineers will contribute to ML and LLMOps pipeline support by building ML data infrastructure for training curation feature engineering and MLflow experiment tracking supporting LLM fine-tuning workflows through corpus curation quality filtering and dataset formatting implementing automated evaluation pipelines for factual accuracy hallucination detection and regression tracking and maintaining production monitoring dashboards for pipeline health model metrics and alerting.

They will also develop agentic data infrastructure by building and maintaining data APIs tool schemas and memory or state stores for autonomous agents implementing agent observability to capture inputs retrieved context tool calls reasoning traces and outputs and maintaining text-to-SQL layers semantic query interfaces and context APIs for conversational AI consumers.

Governance security and data quality responsibilities include implementing role-based and attribute-based access PII detection and masking data classification and audit logging enforcing data contracts and schema governance with automated breaking-change detection and versioned migrations building data quality monitoring for completeness freshness and consistency with automated alerting and root-cause tooling and supporting compliance readiness through audit trails data provenance and regulatory documentation.

Candidates must have five to eight years of data engineering experience and at least two years of production AI ML or LLM-era data infrastructure experience. They should demonstrate proven expertise in building production pipelines at scale in batch and streaming environments with Snowflake and AWS or Azure deep knowledge of Python PySpark Snowflake Delta Lake Kafka and Spark Structured Streaming and hands-on experience with vector stores embedding pipelines and retrieval infrastructure in production RAG environments. Working knowledge of MLOps including MLflow CI/CD for AI automated evaluation and production monitoring along with strong grounding in data governance quality frameworks and compliance-aligned engineering is essential.

Technical skills required include expert-level proficiency in Python SQL PySpark Kafka Delta Lake AWS services such as S3 Glue Kinesis EKS and Redshift Docker Kubernetes GitHub Actions and Snowflake. Strong skills in LangChain LlamaIndex LLM APIs such as OpenAI Bedrock Claude and HuggingFace Pinecone FAISS ChromaDB OpenSearch MLflow FastAPI and Neo4j are expected. Solid skills in CI/CD pipelines CloudWatch Grafana data lineage platforms and MCP along with familiarity with LangGraph prompt engineering RLHF dataset preparation and LLM fine-tuning workflows are desirable.

The technology stack includes Delta Lake PySpark Kafka Spark Structured Streaming Snowflake AWS services such as S3 Glue EKS Bedrock Kinesis Redshift and Lambda Azure Kubernetes Docker Terraform GitHub Actions Jenkins MLflow LangChain LlamaIndex HuggingFace OpenAI AWS Bedrock Claude Pinecone FAISS ChromaDB OpenSearch Neo4j FastAPI Python SQL MCP LangGraph MLOps CI/CD Grafana and CloudWatch.