Senior Data Engineer

Role : Sr. Data Engineer-Databricks

Description

Weare looking for an experienced Senior Databricks Data Engineer with strong hands-on expertise in PySpark, Delta Lake, and cloud data pipelines. The ideal candidate will have experience integrating SAP systems (SAP BDC / SAP Datasphere / SAP ECC / SAP S/4HANA) with Databricks and a solid engineering background.

Responsibilities

• Design, develop, and maintain scalable data pipelines on Databricks using PySpark and Delta Lake.
• Build and optimize ETL/ELT workflows integrating SAP systems with Databricks (using OData, SAP BTP, CDC tools, DB Connectors, or partner solutions).
• Collaborate with AI teams to integrate LLMs and advanced ML models into data workflows.
• Ingest structured and unstructured data from multiple SAP and non-SAP sources into the lakehouse.
• Implement data modeling (Medallion architecture – Bronze/Silver/Gold).
• Optimize Databricks clusters, jobs, workflows, and performance tuning of PySpark workloads.
• Ensure data quality, governance, lineage, and compliance across all pipelines.
• Troubleshoot data issues, analyze performance bottlenecks, and ensure pipeline reliability.
• Prepare technical documentation, deployment scripts, and reusable frameworks.

Required Skills

• 8-12 years of professional experience in data engineering.
• Strong hands-on experience with Databricks, PySpark, Delta Lake, and distributed computing.
• Experience integrating SAP systems with cloud data platforms or Databricks.
• Preferred: SAP BDC / SAP Datasphere / SAP ECC / S/4HANA data model knowledge
• Familiarity with SAP connectors, OData, CDS Views, CPI, or 3rd-party extraction tools
• Expertise in SQL, data modeling, ETL/ELT pipeline development.
• Familiarity with AI/ML concepts, LLMs, Databricks Genie and their integration with data platforms.
• Experience with at least one cloud platform (Azure / AWS / GCP).
• Understanding of CI/CD, DevOps, Git, Databricks Repos, and orchestration tools (Airflow/Azure Data Factory).
• Strong debugging, optimization, and performance-tuning skills.
• Excellent communication and cross-functional collaboration abilities.
• Bachelor's or Master's degree in Engineering, Computer Science, or related field.
• Certifications in Databricks and other relevant data technologies are highly desirable.