Senior Data Engineer

Company Profile

Our client is a global IT services company that helps businesses with digital transformation with offices in India and the United States. It helps businesses with digital transformation, provide IT collaborations and uses technology, innovation, and enterprise to have a positive impact on the world of business.

With expertise is in the fields of Data, IoT, AI, Cloud Infrastructure and SAP, it helps accelerate digital transformation through key practice areas - IT staffing on demand, innovation and growth by focusing on cost and problem-solving.

Job Profile: Senior / Lead Data Engineer

Location: Gurugram

Employment Type:  Full-time

Preferred experience: 5+ years

The Role

We are seeking a highly skilled Senior/Lead Data Engineer with deep expertise in Databricks, Apache Spark, PySpark, distributed data processing, and modern AI/LLM-enabled data platforms. This role is specifically targeted toward engineers who have built production-grade scalable data and AI platforms - not generic ETL developers.

The ideal candidate must demonstrate strong hands-on coding capability, advanced Spark optimization expertise, enterprise-grade CI/CD engineering experience, and practical exposure to LLM evaluation, fine-tuning workflows, vector-space architectures, and ML model evaluation pipelines. Retail domain experience is highly preferred.

Responsibilities

·      Design and build scalable data platforms using Databricks, Spark, Delta Lake, and PySpark.

·      Develop and optimize large-scale data pipelines and distributed processing applications.

·      Perform Spark performance tuning, query optimization, and troubleshooting.

·      Build and manage CI/CD pipelines using GitLab and automation tools.

·      Work on REST APIs, Databricks automation, and deployment workflows.

·      Support AI/ML workflows including LLMOps, RAG pipelines, vector search, and model evaluation.

·      Implement data governance, security, metadata management, and data quality standards.

·      Conduct code reviews, mentor junior engineers, and drive engineering best practices.

·      Collaborate with DevOps, ML, and Business teams to deliver scalable AI and analytics platforms.

Must - Have Qualifications:  

·      5+ years of hands-on experience in Data Engineering with strong enterprise delivery experience.

·      Proven experience as a Senior Data Engineer / Lead Data Engineer in large-scale environments.

·      Expert-level programming experience in Python and PySpark.

·      Experience working on Python IDE-based enterprise development environments.

·      Strong expertise in Apache Spark internals and Spark performance optimization.

·      Hands-on experience with: Databricks, Delta Lake, Spark SQL, Advanced SQL optimization

·      Extensive experience building and maintaining enterprise CI/CD pipelines.

·      Strong expertise in GitLab pipelines and automated deployment workflows.

·      Hands-on experience with Databricks REST APIs.

·      Strong understanding of distributed data processing and scalable architecture design.

·      Experience with LLM evaluation frameworks and ML model evaluation pipelines.

·      Experience supporting or implementing LLM fine-tuning workflows.

·      Experience in ML model evaluation and AI platform engineering.

·      Experience supporting GenAI, RAG, or LLMOps initiatives.

·      Experience with vector-space architectures, embeddings, semantic search, or vector databases.

·      Experience with data governance, metadata management, lineage, and data quality frameworks.

·      Strong experience in scalable data integration and distributed processing frameworks.

Preferred Qualifications:

·      Experience designing reusable framework-based engineering solutions rather than project-specific implementations.

·      Strong analytical, debugging, and problem-solving capabilities.

·      Excellent stakeholder communication and technical leadership skills.

Application Method

Apply on LinkedIn or email your resume to: [Confidential Information]