Technical Lead

Greetings from KPI Partners!

KPI Partners is a 5-time Gartner-recognized consulting firm, specializing in Data Analytics and Digital Transformation. With over 19 years of experience, we are a pioneer in AI consulting. Headquartered in Newark, California, we also maintain regional offices across major U.S. cities including Boston, Charlotte, New York, Houston, Cincinnati, Los Angeles, Atlanta, and Washington, D.C. Our global presence extends offshore to India—with offices in Bangalore, Pune, and Hyderabad—and nearshore through our location in Mexico. We proudly serve over 500 clients worldwide across diverse industries such as Retail & CPG, Manufacturing, Life Sciences, Energy & Utilities, Financial Services, and Insurance.

Job Title: Lead Databricks Engineer (Databricks)

Exp: 8 to 12 years

Location: Bengaluru / Hyderabad / Pune

KPI Partners is seeking a skilled and experienced Lead Databricks Engineer to join our dynamic team. The ideal candidate will have a strong background in Databricks, Pyspark, CI/CD practices, and MLOps. This role will be pivotal in developing data solutions that drive business insights and enhance operational efficiency.

Key Responsibilities:

• Lead the design and implementation of Databricks-based data solutions.
• Develop, test, and deploy data processing pipelines using Pyspark.
• Collaborate with data scientists and analysts to implement MLOps practices for deploying machine learning models.
• Establish and maintain CI/CD pipelines to automate the deployment process and ensure high-quality code delivery.
• Optimize data workflows and improve system performance within the Databricks environment.
• Mentor and guide junior engineers and team members in best practices and emerging technologies.
• Work closely with stakeholders to understand business requirements and translate them into technical specifications.

Required Skills:

• Proficient in Databricks and its ecosystem.
• Strong experience with Pyspark for data processing and analysis.
• Solid understanding of CI/CD practices and tools.
• Familiarity with MLOps concepts and their application in real-world scenarios.
• Excellent problem-solving skills and attention to detail.
• Ability to work collaboratively in a fast-paced, team-oriented environment.

Qualifications:

• Bachelor's degree in computer science, Engineering, or a related field.
• 5+ years of experience in data engineering or a related role, with at least 2 years of experience specifically in Databricks.
• Proven experience in leveraging Pyspark for data processing.
• Experience implementing CI/CD practices in data projects.
• Knowledge of machine learning concepts and MLOps practices is a plus.

Mandatory Skills:

• Databricks
• Pyspark
• CICD
• MlOps
• AWS,Azure