AI Data Engineer

Job Title: AI Data Engineer

Experience: 3–5 Years

Location: Pune

Role Overview:

We are looking for a Databricks and Agentic AI Engineer with 3–5 years of experience to design, develop, and deploy scalable data pipelines and AI agent-based solutions. The role requires strong hands-on experience in Databricks, PySpark, Python, cloud platforms, and modern Agentic AI frameworks.

Key Responsibilities

• Build and optimize large-scale data pipelines using Databricks, PySpark, Spark SQL, Delta Lake, and Unity Catalog.
• Design and implement Agentic AI solutions using LLMs, RAG, tool-calling, multi-agent workflows, and orchestration frameworks.
• Develop production-ready Python components for data processing, AI workflows, APIs, and integrations.
• Work with Databricks Workflows, notebooks, jobs, clusters, Delta Live Tables, and performance tuning.
• Integrate AI agents with enterprise data sources, vector databases, APIs, and cloud services.
• Deploy and monitor solutions on AWS and Azure, following security, scalability, and cost best practices.
• Collaborate with data engineering, analytics, and AI teams to convert business requirements into robust technical solutions.

Required Skills

• 3–4 years of experience in Python, PySpark, and SQL.
• Strong hands-on experience with Databricks, including notebooks, jobs, clusters, Delta Lake, Spark optimization, and data pipeline development.
• Practical experience with Agentic AI, including LLM-based agents, RAG pipelines, prompt engineering, tool use, memory, and workflow orchestration.
• Experience with frameworks such as LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel, or similar.
• Working knowledge of AWS and Azure services for storage, compute, security, and AI integration.
• Understanding of vector databases, embeddings, model APIs, and enterprise AI deployment patterns.

Good to Have

• Experience with Databricks Mosaic AI, MLflow, Model Serving, Feature Store, and Unity Catalog.
• Knowledge of CI/CD, Git, Docker, REST APIs, and MLOps/LLMOps.
• Exposure to Azure OpenAI, AWS Bedrock, or OpenAI APIs.