SENIOR DATA ARCHITECT - Databricks

Job Description Senior Databricks Architect

Position

Senior Databricks Architect

Location

Offshore Bangalore

Domain

Retail / Grocery

Experience

10+ years overall IT experience, with 5+ years in Databricks / Cloud Data Engineering / Data Architecture

Role Overview

We are looking for an experiencedSenior Databricks Architectto lead the architecture and technical execution of a large-scaleTalend-to-Databricks migrationfor a leading Retail/Grocery client.

The architect will be responsible for designing the targetDatabricks Lakehouse architecture, defining migration patterns, guiding the data engineering team, and ensuring successful modernization of existing Talend-based ETL/ELT workloads onto Databricks.

The role requires strong hands-on expertise inDatabricks, Apache Spark, SQL, cloud data platforms, data architecture, ETL modernization, and migration strategy, along with a good understanding ofRetail/Grocery business processes and data domains.

Key Responsibilities

Architecture & Solution Design

• Lead the end-to-end architecture for migration of Talend ETL/ELT workloads to Databricks.
• Define the target-stateDatabricks Lakehouse architecture, including data ingestion, transformation, storage, orchestration, governance, and consumption layers.
• Design scalable, secure, highly available, and cost-optimized data solutions.
• Define architecture standards, design patterns, coding standards, and best practices for Databricks development.
• Establish migration frameworks and reusable patterns for converting Talend jobs into Databricks/Spark-based pipelines.
• Evaluate existing Talend jobs and determine appropriate migration strategies, including re-platforming, re-engineering, consolidation, or retirement.
• Provide technical leadership for complex data pipelines and integration scenarios.
  
  

Talend to Databricks Migration

• Analyze existing Talend workflows, jobs, mappings, dependencies, schedules, and data transformations.
• Develop migration strategies for batch and incremental data processing workloads.
• Translate Talend transformations and business rules intoPySpark/SQL/Databricksimplementations.
• Identify opportunities to simplify and optimize legacy ETL processes during migration.
• Define data reconciliation and validation strategies to ensure parity between Talend and Databricks outputs.
• Establish migration sequencing based on business criticality, dependencies, complexity, and risk.
• Support migration of high-volume and business-critical data pipelines.
  
  

Databricks & Data Engineering

• Design and implement solutions usingDatabricks, Apache Spark, Delta Lake, PySpark, and SQL.
• DesignMedallion Architecture (Bronze, Silver, Gold)and appropriate data processing patterns.
• Implement incremental processing, CDC, SCD Type 1/Type 2, data quality, error handling, and audit frameworks.
• Optimize Spark workloads, Delta tables, SQL queries, partitioning, clustering, and job execution.
• LeverageDelta Live Tables / Lakeflow capabilities, Workflows, Unity Catalog, and other Databricks platform services where appropriate.
• Design data pipelines integrating structured and semi-structured data from databases, files, APIs, and enterprise applications.
• Establish monitoring, logging, alerting, operational support, and performance-management patterns.
  
  

Cloud & Integration

• Strong experience with at least one major cloud platform, preferablyAzure or AWS.
• Design integration between Databricks and cloud storage, databases, messaging platforms, APIs, and enterprise applications.
• Experience with technologies such asADLS/S3, Azure Data Factory, Kafka, REST APIs, relational databases, and cloud-native servicesis highly desirable.
• Define secure connectivity, networking, secrets management, and access-control patterns.
  
  

Retail / Grocery Domain

• Work closely with business and technology stakeholders to understand Retail/Grocery data requirements.
• Experience with retail data domains such as:
• Product / Item
• Store / Location
• Customer / Loyalty
• Sales / Transactions
• Pricing
• Promotions
• Inventory
• Supply Chain
• Vendors / Suppliers
• Orders
• Forecasting
• Merchandising
• Understand retail-specific challenges such as high-volume transaction processing, store-level data, product hierarchies, promotions, pricing, inventory movements, and customer analytics.
  
  

Databricks