Role & Responsibilities
You are the technical lead for a strategic project on LLM training. You own the complete projectfrom data mixture strategy to the final model evaluationensuring the project meets the client's domain-specific performance requirements within strict compute and data-privacy constraints.
Key Responsibilities
- Lead the Training Roadmap : Define the stages of the LLM lifecycle : Continuous Pre-training (CPT) on domain-specific data, SFT, and alignment (DPO/PPO/RLHF).
- Architect Distributed Training Systems : Design the infrastructure for multi-node, multi-GPU training. You will make the call on using the right infrastructure.
- Data Mixture & Curriculum Design : Own the data strategy. You will decide the fine tuning between general-purpose data and domain-specific corpora to prevent catastrophic forgetting.
- Compute Budget & Optimization : Manage infrastructure optimize training efficiency (TFLOPS/GPU) and implement checkpointing strategies to ensure zero loss of progress during hardware failures.
- Evaluation Framework Design : Build a comprehensive, sovereign evaluation suite (beyond standard benchmarks) that measures the model's performance on the clients specific data.
- Direct Customer Delivery : Act as the primary technical point of contact for the customers team, translating their requirements into training approach and solution design.
Required Technical Skillset
- Distributed Training & Infrastructure :
- 3D Parallelism Mastery : Hands-on implementation of Data, Tensor, and Pipeline parallelism for models exceeding 70B parameters.
- Backend Optimization : Expert tuning to resolve bottlenecks in multi-node GPU clusters.
- Memory & Fault Tolerance : Mastery of ZeRO (Stages 13) and activation checkpointing; building automated, elastic training workflows for rapid failure recovery.
- LLM Training Tricks & Stability :
- Convergence Engineering : Expertise in mitigating loss spikes, adaptive gradient clipping, and sophisticated LR schedules.
- Numerical Precision : Hands-on experience with Mixed Precision (BF16/FP16) and FP8 workflows, including dynamic loss scaling management.
- Advanced Optimization : Leveraging Grokking techniques and monitoring gradient health to adjust curriculum mid-run.
- Data Strategy : Implementing curriculum learning and domain-specific mixture ratios to prevent catastrophic forgetting during Continuous Pre-training (CPT).
- Efficiency & Quantization Tricks :
- Advanced PEFT : Implementation of QLoRA, DoRA, and Rank-Stabilized LoRA to ensure fine-tuning performance matches full-parameter training.
- Inference-Aware Training : Utilizing Grouped Query Attention (GQA) and Multi-Query Attention (MQA) during training to optimize models for high-throughput deployment.
Required Qualifications
- Experience : 15+ years in technology, with dedicated experience in design and development of transformer based AI/ML-based systems.
- GCP Mastery : Deep expertise in designing AI/ML solutions on GCP, specifically leveraging the Vertex AI ecosystem. A Google Cloud Professional Machine Learning Engineer certification is highly desirable.
- LLM Expertise : Hands-on experience architecting and delivering large-scale, production-grade AI systems, Proven experience in building and productionizing applications using Large Language Models (LLMs), including expertise in vector databases and model fine-tuning.
(ref:hirist.tech)
