Databricks Engineer

Role: Databricks Engineer

Location: Pune

Type of Employment: Permanent

About CRISIL:

CRISIL is a global analytical company providing ratings, research, and risk and policy advisory services. We are India's leading ratings agency. We are also the foremost provider of high-end research to the world's largest banks and leading corporations. CRISIL's majority shareholder is S&P Global.

Division: Global Analytical Centre (GAC)

Role Overview:
We are looking for an experienced Databricks Engineer to design, develop, and maintain scalable data engineering solutions using the Databricks Lakehouse platform. The role will involve building robust ETL/ELT pipelines, developing data transformation frameworks, optimizing Spark workloads, and integrating data from multiple enterprise sources.
The ideal candidate should have strong hands on experience with Databricks, Apache Spark, PySpark, SQL, Python, Delta Lake, and cloud data platforms, along with good understanding of data architecture and data governance.

Key Responsibilities:

• Design and develop scalable data pipelines using Databricks, Apache Spark and PySpark.
• Build and maintain ETL/ELT pipelines for structured and semi-structured data.
• Develop data transformation and processing workflows using Python, PySpark and SQL.
• Implement and manage Delta Lake tables, including Delta Live Tables / Lakeflow pipelines where applicable.
• Work with Databricks Workflows/jobs for pipeline orchestration and scheduling.
• Integrate data from databases, APIs, files, cloud storage and other enterprise systems.
• Implement data quality, validation, reconciliation and monitoring frameworks.
• Optimize Spark jobs, SQL queries and Databricks workloads for performance and cost.
• Implement appropriate partitioning, caching, clustering and file-management strategies.
• Develop incremental data processing and Change Data Capture (CDC) solutions.
• Work with cloud platforms such as Azure, AWS or GCP.
• Implement security, access controls and data governance in collaboration with platform teams.
• Troubleshoot production data pipelines and resolve performance and data-quality issues.
• Collaborate with Data Scientists, Business Analysts, Application teams and Technology teams.
• Participate in code reviews, technical design discussions and deployment activities.
• Follow Agile development practices and enterprise SDLC standards.
• Maintain technical documentation, data lineage and operational runbooks.
  

Required Technical Skills:

• Databricks & Big Data
• Strong hands-on experience with Databricks
• Apache Spark
• PySpark

Qualification and Essential Skills:

• Bachelor's degree in Computer Science, Software Engineering or a related field.
• Experience as a Python Developer / SQL with a strong portfolio of projects.
• Comprehensive English communication skills.
• Good interpersonal and decision-making skills.
• Ability to work effectively in a team-oriented, global environment with various business partners.
• Well-organized with great attention to detail.
• Ability to multitask and stay poised in a fast-paced, high-pressure environment while also ensuring the highest quality output.
• Flexible, self-starter who is willing to take the initiative and drive tasks to completion with strong execution.
• Must possess strong prioritization and influencing skills, experience in leading highly valued projects.

Eligibility Criteria:

• Not appeared for CRISIL test / interviews in the last six months
• Familiar with Tableau, and PowerBI is required.
• Certification in Databricks, Cloud technologies, AWS, Agile Methodology and Project Management will be added advantage.