Widely considered to be one of the technology world's most desirable employers, NVIDIA is an industry leader with groundbreaking developments in High-Performance Computing, Artificial Intelligence and Visualization. The GPU, our invention, serves as the visual cortex of modern computers and is at the heart of our products and services. GPU deep learning ignited modern AI - the next era of computing - with the GPU acting as the brain of computers, robots, autonomous cars and conversational AI that can perceive and understand the world. Today, we are increasingly known as the AI computing company. We're looking to grow our company, and build our teams with the smartest people in the world. Join us at the forefront of technological advancement.

NVIDIA is looking for an Inference Optimization architect to accelerate and scale our Speech AI models & improve the experience of millions of customers. You will focus on reducing inference latency, improving throughput, and optimizing resource utilization across our AI infrastructure. If you're creative & passionate about solving real world conversational AI problems, come join our Speech AI Engineering team.

What you'll be doing:

Optimize Inference Performance: Improve streaming latency and throughput through advanced batching strategies, encoder caching, and multi-threaded pipeline optimizations
Model Compression: Implement techniques including quantization, pruning, and knowledge distillation.
Benchmarking: Profile and benchmark models to identify and resolve performance bottlenecks. GPU profiling and debugging using Nsight Systems and Nsight Compute
Hardware Acceleration: Develop custom kernels and leverage hardware acceleration (CUDA, TensorRT, etc.).
Infrastructure Design: Design and implement efficient serving infrastructure for Speech models at scale.
Collaboration: Work alongside Model researchers to transition models from research to production readiness.
Cross-Platform Optimization: Optimize inference across diverse GPUs platforms (data centre, edge devices).
Tooling: Build frameworks for automated model optimization pipelines.
Resource Management: Monitor and improve inference costs and resource utilization in production.

What we need to see:

Masters or BE/BTech in Computer Science, computer architecture, or related field
10+ years of total experience & 5+ years on performance optimizations of Deep learning model inference
Experience with inference pipelines for LLM, Speech Recognition & Speech Synthesis
CUDA kernel development: thread blocks, shared memory, synchronization
Model inference optimization: batching, dynamic shapes, latency tuning
Model serving and deployment: Triton, TorchServe, TensorRT, TRT-LLM, vLLM
Model optimization techniques: quantization, pruning, distillation
Computer architecture & Operating systems: processes, threads, scheduling, memory management
Solid understanding of modern model architectures (Transformers, CNNs, RNNs)

Ways to stand out from the crowd:

Publications or contributions to open-source projects like pytorch/jax/triton-lan
Experience with embedded systems or edge deployment
Strong collaborative and interpersonal skills, specifically a proven ability to effectively guide and influence within a dynamic matrix environment

NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

Inference Optimization Architect, Speech AI