Databricks Architect

Architect - Databricks (Cloud Data)

Job Title: Architect - Databricks (Cloud Data)

Primary skills: Databricks, PySpark, and architecture solutions.

Secondary skills: Unity Catalog, StarBurst, Azure DevOps, CICD, and orchestration layers.

Mode of work: Work from office

Experience: 13+ Years of Experience

Responsibilities:

· Lead migration from Native Spark to Databricks.

· Architect migration to Databricks.

· Build Data Governance solutions using tools like Unity Catalog and StarBurst.

· Develop orchestration workflows using Databricks and ADF.

· Implement CICD pipelines for Databricks in Azure DevOps.

· Process near-real-time data through Auto Loader and DLT pipelines.

· Design secure, scalable infrastructure for Delta Lake and Spark SQL.

· Optimize Databricks environments for cost-effectiveness.

· Extract complex business logic from on-prem solutions like SSIS, Informatica, Vertica, etc., into PySpark.

· Build analytical dashboards on top of the Databricks Lakehouse. Requirements - Must Have:

· 13+ years of overall experience, with at least 4+ years as a solution architect.

· Experience with Databricks, PySpark, and modern data platforms.

· Expertise in cloud-native data platform architecture and solution design.

· Familiarity with Agile/Scrum methodologies.

Experience in pre-sales and in the P&C insurance domain is an advantage.