Principal Databricks Solution Architect

About Palni and SmartBots

Palni is a digital transformation and technology services firm with 500+ technical, domain and business professionals. Its capabilities span data engineering, AI and automation, cloud transformation, Salesforce, ServiceNow, customer experience and engineering services. Palni is a growing Databricks partner serving enterprises across industries.

SmartBots is Palni's wholly owned Agentic AI and Generative AI arm. Together, Palni and SmartBots help clients move from governed data foundations to analytics, AI agents and intelligent applications.

Role Purpose and Leadership Mandate

This role is Palni's senior technical authority for Databricks, combining customer advisory, solution architecture, pre-sales, delivery governance and hands-on technical leadership. The architect will also help build the practice by defining standards and offerings, mentoring architects and engineers, developing reusable assets and strengthening collaboration with Databricks.

·  Builder and talent multiplier: creates teams, standards and reusable capability rather than dependency on one expert.

·  Trusted customer advisor: listens deeply, challenges constructively and communicates technical decisions in business language.

·  Commercially pragmatic architect: balances customer value, scalability, security, delivery feasibility, cost and time to outcome.

·  Continuous learner: keeps pace with the evolving Data + AI platform and raises the capability of the wider team.

Key Responsibilities

Customer Architecture and Pre-sales

·  Lead executive and technical discovery workshops; translate business outcomes and non-functional requirements into Data + AI roadmaps and target architectures.

·  Architect end-to-end solutions covering ingestion, batch and streaming pipelines, lakehouse storage, governance, data warehousing, analytics, ML/GenAI and operational applications.

·  Define phased migrations from legacy warehouses, Hadoop, cloud data platforms and ETL estates to Databricks.

·  Lead technical solutioning for proposals, demonstrations, RFP/RFI responses, estimates and proofs of value; clearly articulate business value and architectural trade-offs.

·  Collaborate with sales, delivery, cloud partners and Databricks account teams on joint pursuits and co-sell readiness.

Delivery Governance and Practice Building

·  Provide architecture governance from pursuit through production, including design, security, performance, cost and operational-readiness reviews.

·  Guide CI/CD, infrastructure as code, environment strategy, testing, observability, release management and production support.

·  Define the practice roadmap, skills matrix, certification plan and hiring profile; mentor architects, data engineers and technical leads.

·  Create reusable reference architectures, discovery and estimation templates, demos, accelerators and technical playbooks.

Databricks Platform Expertise

Core Lakehouse, Data Engineering and Governance

·  Delta Lake and lakehouse design: ACID transactions, schema evolution, Change Data Feed, optimization and Medallion or domain data-product patterns.

·  Unity Catalog: catalogs/schemas, access controls, managed and external locations, lineage, discovery, audit, federation and Delta Sharing.

·  Lakeflow Connect, Lakeflow Spark Declarative Pipelines and Lakeflow Jobs for ingestion, batch/streaming transformation, data quality, orchestration and observability.

·  Apache Spark, PySpark, SQL and Structured Streaming, including workload sizing, optimization and production troubleshooting.

·  Databricks SQL, serverless warehouses, query optimization, AI/BI dashboards and Genie for governed natural-language analytics.

AI, Applications and Platform Engineering

·  Lakebase architecture for PostgreSQL-compatible transactional workloads, application data and agent state, with appropriate integration to the lakehouse.

·  Mosaic AI and MLflow: model lifecycle, evaluation, serving, Vector Search, RAG, AI Gateway, monitoring and governed agentic patterns. Familiarity with Agent Bricks is valuable.

·  Databricks Apps and secure Data + AI application patterns, including the ability to separate operational, analytical and agentic workload requirements.

·  Databricks Asset Bundles, Terraform, Git-based CI/CD, system tables, platform observability, serverless/compute policies and FinOps.

·  Azure architecture is preferred; AWS or Google Cloud experience is valuable. Familiarity with Kafka, ADF, dbt, Snowflake, Synapse or Power BI is advantageous.

Required Experience, Qualifications and Certifications

·  10+ years in data engineering, data architecture, analytics or cloud data platforms, including at least 3+ years of substantial Databricks experience.

·  Demonstrated success architecting and delivering enterprise-scale Databricks solutions in production—not only isolated notebooks or pipelines.

·  Strong pre-sales capability across discovery, technical storytelling, estimates, proposals, demonstrations, proofs of value and senior customer presentations.

·  Strong SQL and Python/PySpark skills, with the ability to review implementations, troubleshoot designs and guide engineering teams.

·  Experience leading architecture governance and multidisciplinary teams; prior practice-building, hiring, mentoring or reusable-IP responsibility is strongly preferred.

·  Professional-level Databricks certification is strongly preferred at joining. An otherwise strong candidate must complete an agreed professional certification within six months.

·  Excellent communication, executive presence and structured problem-solving; able to explain technical and commercial trade-offs clearly.

·  Experience in consulting, a systems integrator or a Databricks partner is preferred, particularly in BFSI, manufacturing, healthcare/life sciences or retail.

·  Willingness to work from Palni's Hyderabad office and travel for customer or partner engagements when required.

The Opportunity

This is an opportunity for a senior Databricks architect to shape a growing practice—not simply inherit one. The successful candidate will influence architecture standards, customer pursuits, delivery quality, team development, partner readiness and Palni's differentiated Data + AI offerings with SmartBots.