Role Summary
We are looking for a visionary Senior Data Architect to lead the evolution of our data ecosystem. You will be responsible for designing and implementing a high-performance Hybrid Lakehouse architecture that leverages Databricks for massive-scale processing and Vertex AI to power our next generation of Generative AI agents. This is a hands-on leadership role where you will move from architectural whiteboarding to deploying production-grade PySpark pipelines and LLM orchestration.
Key Responsibilities
- Design and maintain a scalable GCP data infrastructure using a Medallion Architecture (Delta Lake) and BigQuery.
- Build, tune, and deploy Generative AI solutions using Vertex AI (Gemini, Search & Conversation), including RAG pipelines and Agentic workflows.
- Lead the development of complex ETL/ELT pipelines using PySpark on Databricks, ensuring high data quality and low latency.
- Implement enterprise-wide data governance and security via Unity Catalog and GCP Dataplex.
- Seamlessly integrate Databricks-processed data with Vertex AI for model training and real-time inference.
- Set the standard for LLMOps and DataOps, conducting code reviews and mentoring junior engineers.
Technical Requirements (The Must-Haves)
- 10+ Years Total Experience in Data Engineering, Data Architecture, or Software Engineering.
- 5+ Years of Deep architectural experience in Google Cloud Platform (GCP).
- GCP Ecosystem: 5+ years of experience with BigQuery, GCS, Cloud Composer, and Pub/Sub.
- Vertex AI Mastery: Proven hands-on experience with Vertex AI Pipelines, Model Garden, and Vector Search. You must be able to demonstrate how you've deployed an LLM into production.
- Databricks & PySpark: Expert-level proficiency in PySpark and managing Databricks environments (Clusters, Workflows, Unity Catalog).
- Languages: Expert in Python and SQL.
- DevOps: Experience with Terraform for Infrastructure as Code (IaC) and CI/CD for ML (MLflow).
Preferred Qualifications
- Certifications: Google Cloud Professional Data Engineer or Professional Machine Learning Engineer.
- GenAI Portfolio: Experience building custom AI Agents or implementing complex Retrieval-Augmented Generation (RAG) architectures.
- Industry Context: Experience handling large-scale datasets (Petabyte scale) in a regulated environment.
