Data Engineer - Lead

Location - Bengaluru ( Hybrid )

Experience - 7+ years

We are seeking a Lead Data Engineer with strong expertise in Databricks, Python, and modern data pipeline architectures to design, build, and optimize scalable data platforms. The role focuses on developing robust ELT/ETL pipelines, implementing data lake architectures, and ensuring efficient data transformations to support analytics and AI-driven use cases. The ideal candidate will have hands-on experience with distributed computing concepts, SQL-based transformations, and cloud-scale data engineering practices, along with strong problem-solving and mentoring abilities.

Key Responsibilities

• Design, build, and maintain scalable data pipelines and ingestion frameworks using Python, SQL, and Databricks.
• Develop and optimize ELT/ETL workflows to ingest and transform data from multiple enterprise systems.
• Implement Python-based data transformation pipelines using libraries such as Pandas, Polars, or Delta Live Tables (DLT).
• Work with Databricks platform capabilities for distributed data processing and large-scale data transformations.
• Build and maintain integrations using ELT tools such as cData, Hevo, MuleSoft, or Fivetran.
• Design and maintain data lake architectures, implementing Medallion architecture (Bronze, Silver, Gold layers).
• Apply strong data modeling and schema design principles to support analytics and reporting workloads.
• Monitor, debug, and troubleshoot data pipelines and workflows to ensure reliability and performance.
• Optimize data transformations and query performance for large-scale datasets.
• Document data pipelines, transformations, and architecture designs to support maintainability and governance.
• Collaborate with data analysts, BI teams, and stakeholders to deliver high-quality data solutions.
• Mentor junior engineers and support team productivity and delivery timelines.

Essential Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or related field.
• Strong hands-on experience with Databricks platform and distributed data processing.
• Proficiency in Python for data engineering, including libraries such as Pandas, Polars, or Delta Live Tables (DLT).
• Strong experience with SQL for data transformation and analysis.
• Deep understanding of ETL/ELT principles and modern data pipeline architectures.
• Experience designing and implementing data ingestion pipelines.
• Strong understanding of distributed compute and storage concepts.
• Experience with data lake architecture and Medallion architecture patterns.
• Solid knowledge of data modeling, schema design, and data relationships.
• Experience with pipeline orchestration, monitoring, and troubleshooting.
• Proficiency with Git version control and modern engineering workflows.
• Exposure to Salesforce or NetSuite data schemas and integrations.

Nice to Have Skills

• Experience using dbt for data transformations.
• Knowledge of data validation frameworks such as Great Expectations.
• Familiarity with vector databases such as Weaviate.
• Understanding of embeddings and LLM-based data workflows using tools like Ollama or OpenAI APIs.
• Experience working with Agile methodologies or Scrum environments.
• Exposure to commercial or business-facing data engineering environments.

Soft Skills

• Strong problem-solving and analytical thinking abilities.
• Excellent communication skills for collaborating with technical and business stakeholders.
• Ability to design scalable solutions and make architectural decisions.
• Strong documentation and knowledge-sharing mindset.
• Experience mentoring team members and supporting delivery timelines.
• Ability to work effectively in fast-paced, agile data engineering environments.

Benefits

• Comprehensive benefits package including health insurance, paid time off, and professional development opportunities.

About Potentiam

Potentiam is a global provider of highly qualified professionals to European SMEs from our offices in Romania, South Africa and India. Potentiam works with clients in finance, energy, leisure, marketing, business services and technology industries, providing technical, professional multi- lingual highly motivated staff, most of whom have had experience of working for international companies. Staff cover a wide range of roles from accounting, marketing, data management, HR, sales/account management, engineering, technology, and operations. Potentiam manages our staff's career development and personal development training, all infrastructure, HR and payroll with our clients directly managing day-to-day staff responsibilities and role training and development.

If interested please apply here, if you have any questions regarding the role, please feel free to write to [Confidential Information]

Data Privacy Notice

The personal information you provide during the application and recruitment process will be used solely for recruitment purposes, in accordance with our data protection policies.

For any questions regarding data processing related to HR activities, please contact at [HIDDEN TEXT]

All data shared with third parties complies with applicable confidentiality and retention requirements.