Data Engineer (Azure Databricks & PySpark)

Data Engineer (Azure Databricks & PySpark)

• Primary Skills: Azure Data bricks
• Secondary Skills: Python, PySpark
• Location – Bangalore
• Experience – 6 to 12 years
• Interview Mode - Virtual
• Shift timing – General Shift
• Notice period – Immediate Joiner or October 2nd Week LWD candidates

Data Engineer (Azure Databricks & PySpark)

Job Title: Data Engineer – Azure Databricks (ADB) & PySpark

Job Description:

We are looking for a Data Engineer with strong experience in Azure Databricks (ADB) and PySpark to design, develop, and optimize large-scale data processing solutions. The candidate will be responsible for building scalable ETL/ELT pipelines, transforming data, and supporting analytics and reporting requirements.

Key Responsibilities:

• Develop and maintain data pipelines using Azure Databricks and PySpark.
• Design and implement ETL/ELT processes for data ingestion and transformation.
• Work with structured and unstructured data from multiple sources.
• Optimize Spark jobs for performance and scalability.
• Develop and maintain data models for analytics and reporting.
• Collaborate with business stakeholders and technical teams to understand data requirements.
• Ensure data quality, integrity, and governance standards are maintained.

Required Skills:

• Hands-on experience with Azure Databricks (ADB).
• Strong proficiency in PySpark and Spark frameworks.
• Good experience in SQL and data transformation.
• Knowledge of Azure Data Lake Storage (ADLS).
• Understanding of Data Warehousing and ETL/ELT concepts.
• Experience in performance tuning and optimization of Spark applications.
• Strong problem-solving and analytical skills.

Preferred Skills:

• Experience with Azure Data Factory (ADF).
• Knowledge of Delta Lake, Medallion Architecture, and Data Lake concepts.
• Experience working in Banking, Finance, Insurance, or Retail domains.
• Familiarity with Git and CI/CD practices.