Company Description
ThreatXIntel is a growing Cybersecurity, IT Staffing, and Consulting company delivering end-to-end technology and security solutions.
We are hiring for our corporate client. ThreatXIntel is the official hiring partner for this requirement.
Job Overview
We are seeking a highly skilled Databricks Architect to lead the design and implementation of enterprise-scale Lakehouse data platforms.
The ideal candidate will have deep expertise in Databricks, Apache Spark, Delta Lake, and cloud-native architectures, with proven experience delivering end-to-end data engineering solutions across batch and real-time pipelines.
This role requires strong architectural leadership, hands-on development capability, and stakeholder collaboration.
Key Responsibilities
Architecture & Design
- Architect and implement enterprise-grade Lakehouse solutions using Databricks
- Design scalable cloud-based data platforms integrating multiple data sources
- Define data architecture standards, governance, and best practices
Data Engineering & Pipelines
- Build end-to-end ETL/ELT pipelines using PySpark, Scala, and SQL
- Develop batch and real-time streaming pipelines using Spark
- Design incremental loading frameworks and metadata-driven ingestion pipelines
Databricks Ecosystem
- Implement and manage:
- Delta Live Tables
- Autoloader
- Structured Streaming
- Databricks Workflows
- Integrate with orchestration tools like Apache Airflow
Data Modeling & Governance
- Design 3NF, dimensional models, and enterprise data warehouse solutions
- Implement data quality frameworks and governance standards
- Manage Unity Catalog with fine-grained security and access control
Performance Optimization
- Optimize Spark jobs, pipelines, and compute resources
- Ensure scalability, reliability, and cost efficiency
DevOps & CI/CD
- Implement CI/CD pipelines and deployment strategies
- Drive DevOps best practices for data engineering
Leadership & Collaboration
- Provide technical leadership and architectural guidance
- Collaborate with stakeholders to translate business needs into scalable solutions
Required Skills (Mandatory)
- Deep expertise in Databricks and Lakehouse architecture
- Strong experience with Apache Spark (batch & streaming)
- Hands-on experience with Delta Lake
- Strong programming skills in Python, PySpark, Scala, and SQL
- Experience with distributed data processing systems
- Strong experience in data modeling and data warehousing concepts
- Experience with real-time data processing architectures
- Hands-on experience with Unity Catalog and data governance
