Network Engineer

Job Purpose

Deliver and operate ultra‑low‑latency, high‑bisection network fabrics for GPU training at 10K GPU scale with deterministic performance.

Roles & Responsibilities

• Implementation

Build IB (HDR/NDR) east‑west fabrics and Ethernet north‑south connectivity; leaf‑spine with ECMP.

Deploy and tune RoCE v2, PFC/ECN, MTU, buffer profiles; IB routing/credit/timers; SHARP/UCX offloads where applicable.

• Operations

Day‑2 fabric health (congestion hotspots, slow‑drain, link flaps); automate config with Ansible/Terraform.

Lifecycle of switch OS/firmware; link‑level optics and cable qualification; inventory and spares.

• Performance & Optimization

Validate NCCL collectives performance across pods; troubleshoot all‑reduce bottlenecks and imbalance.

Implement QoS classes for training vs inference vs control‑plane.

• Reliability & Incident

Lead P0/P1 critical incident response for NVIDIA GPU infrastructure, including InfiniBand, RoCE, Ethernet fabric and AI/ML cluster environments.

Troubleshot performance bottlenecks involving InfiniBand fabrics, RoCE networks, congestion management, NCCL communications, GPUDirect RDMA, and distributed AI training workloads.

Led post-incident reviews and implemented automation, monitoring enhancements, and architectural improvements to increase overall cluster availability, stability, and performance.

Create runbooks for path failures, fabric partitions, and credit starvation; design graceful degradation.

• Security & Compliance

BGP/EVPN multi‑tenancy, ACLs, role‑based access to fabric managers; config signing/change control.

Experience & Educational Requirement

BE/B-Tech or equivalent with Computer Science or Electronics & Communication

Certifications:- NVIDIA Networking; CCNP/CCIE DC; JNCIP‑DC / Arista ACE.

RELEVANT EXPERIENCE

• 7–15 years DC networking; deep InfiniBand (HDR/NDR) and RoCE tuning at scale; GPU cluster comms patterns.

Tools / Tech

NVIDIA UFM/NetQ (or equivalent), Spectrum/Quantum families, Cumulus/Onyx/SONiC; iperf/NCCL‑tests; sFlow/telemetry; Ansible/Terraform.