Software Engineer

Role description
Python + Gen AI + Databricks Engineer
Role Summary

We are seeking a highly skilled Python + Gen AI + Databricks Engineer to design, build, and deploy scalable data and AI solutions. The role involves developing data pipelines, advanced analytics, and Generative AI applications on cloud-based platforms using Databricks, Python, and modern LLM frameworks.

Key Responsibilities

• Develop and maintain data pipelines and analytics workflows using Databricks (Spark, PySpark)


• Build, deploy, and optimize Python-based data engineering and machine learning solutions


• Design and implement Generative AI use cases such as chatbots, summarization, content generation, and retrieval-augmented generation (RAG)


• Integrate LLMs (OpenAI, Azure OpenAI, Hugging Face, etc.) into enterprise applications


• Perform data preprocessing, feature engineering, and model validation


• Optimize data workflows for performance, scalability, and cost efficiency


• Collaborate with business stakeholders, data scientists, and product teams to translate requirements into technical solutions


• Ensure data quality, security, and compliance standards are met


• Document solutions and provide production support where required

Required Skills & Qualifications
Technical Skills

• Strong proficiency in Python (data structures, OOP, APIs, performance tuning)


• Hands-on experience with Databricks:
  
  
  
  • Apache Spark / PySpark
  
  
  • Delta Lake
  
  
  • Databricks workflows and jobs
  
  
  
  


• Experience with Generative AI / LLMs:
  
  
  
  • Prompt engineering
  
  
  • Fine-tuning or adapter-based approaches (LoRA, adapters)
  
  
  • RAG architectures
  
  
  
  


• Experience with ML/AI frameworks: LangChain, LlamaIndex, TensorFlow, PyTorch (any)


• Strong SQL skills and experience with large datasets


• Experience with cloud platforms (Azure preferred; AWS/GCP acceptable)

Data & MLOps

• Model deployment and monitoring


• CI/CD for data and ML pipelines


• Experiment tracking and version control


• Familiarity with MLflow is a plus

Preferred Qualifications

• Experience with Azure Databricks and Azure OpenAI


• Knowledge of vector databases (FAISS, Pinecone, Weaviate)


• Exposure to real-time or streaming data (Kafka, Event Hubs)


• Understanding of data governance and security concepts


• Prior experience in retail, BFSI, healthcare, or large-scale enterprise analytics

Education

• Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or related field

Experience

• 4–8 years of experience in data engineering, analytics, or AI development


• At least 1–2 years of hands-on experience with Generative AI or LLM-based solutions