Role Title: HPC Engineer - Network
Location: India (Must align with Client Time Zone)
Employment Type: Full-Time
About The Role
The
HPC Engineer - Network acts as the primary execution engine for the connectivity that binds the AI Factory together. While the Domain Architect designs the topology and the Senior Engineer directs the squad, you are the Builder responsible for configuring, stabilising, and tuning the high-speed interconnects. You are a doer who is as comfortable debugging a flapping InfiniBand link via CLI as you are pushing a configuration update to fifty switches using Ansible.
As a System Integrator, we design and deliver bespoke, high-scale AI factories. In this role, you will move beyond standard enterprise networking (campus/wifi) to execute the deployment of lossless, high-bandwidth fabrics for
NVIDIA SuperPOD, NVIDIA BasePOD, and Cisco AI Factory environments.
In this role, you will operate with a
100% focus on Delivery, executing the Low-Level Designs (LLD) assigned by your Squad Lead. You will own the Network in the critical Compute-Network-Storage triad, ensuring that the GPU compute nodes can communicate at line rate without congestion.
CRITICAL REQUIREMENT: This role typically operates on
Shift Hours to align with the onshore client's time zone (e.g., early shifts for Australian clients, or split shifts for European clients).
Key Responsibilities
- Fabric Configuration & Provisioning
- Switch Configuration: Execute the configuration of high-performance switches (NVIDIA Quantum InfiniBand, NVIDIA Spectrum-X Ethernet, Cisco Nexus) using defined templates and automation.
- NetDevOps Execution: Run and maintain Ansible playbooks to push configurations, update firmware (Cumulus/NX-OS), and enforce compliance across the fabric.
- Subnet Management: Configure and tune NVIDIA Unified Fabric Manager (UFM) to ensure optimal routing and fault tolerance.
- Host Networking: Assist the Compute team in configuring host-side adapters (ConnectX SuperNICs, BlueField DPUs) to ensure correct IP addressing, MTU, and driver parameters.
- Validation & Performance Tuning
- Link Validation: Verify physical connectivity and link health using tools like ibstat, ibdiagnet, and ethtool to identify faulty cables or transceivers immediately after installation by Field Engineers.
- Performance Testing: Execute network-specific benchmarks (e.g., ib_write_bw, ib_send_bw, NCCL-tests) to validate that the fabric is delivering full bi-sectional bandwidth and low latency.
- Congestion Control: Implement and tune Quality of Service (QoS) settings, including Priority Flow Control (PFC), Explicit Congestion Notification (ECN), and Data Center Quantized Congestion Notification (DCQCN) to prevent packet loss and optimize throughput in RoCEv2 environments.
- Operations & Support
- Fabric Telemetry: Configure monitoring agents (e.g., Prometheus node exporters, UFM Telemetry) to visualize traffic flows and detect tail latency issues.
- Ticket Resolution: Handle L2 support tickets for network issues, such as Node isolation, Slow All-Reduce operations, or Fabric flapping.
- Lifecycle Management: Execute firmware upgrades on switches and DPUs during maintenance windows, ensuring strict compatibility with the NVIDIA OFED stack.
,
Technical Competencies
Essential Skills
High-Performance Networking:
- InfiniBand Mastery: Deep operational knowledge of NVIDIA Quantum InfiniBand switches, cable types (NDR/HDR), and troubleshooting commands.
- AI Ethernet (RoCEv2): Solid understanding of RDMA over Converged Ethernet (RoCEv2), including the configuration of PFC and ECN on switches (Spectrum/Arista/Cisco/Juniper).
- Fabric Management: Experience with NVIDIA UFM (Unified Fabric Manager) for managing large-scale fabrics.
Automation & Tools:
- NetDevOps: Proficiency in Ansible for network automation (e.g., ansible-networking collections).
- Linux Networking: Comfortable navigating Linux CLI to troubleshoot host-side networking (ip link, tcpdump, sysctl tuning).
- Protocol Knowledge: Practical implementation skills in BGP, EVPN, and VXLAN for multi-tenant AI clouds.
Desirable Experience
- Cisco AI Integration: Experience with Cisco Nexus Dashboard or Cisco 8000 series in an AI context.
- DPU Configuration: Experience configuring NVIDIA BlueField DPUs and working with the DOCA framework.
- Optical Networking: Ability to interpret transceiver signal levels (DOM/DDM) to diagnose Layer 1 optical faults.
- AI Fabric Orchestration: Experience deploying and managing AI clusters using Netris or Cisco Nexus Hyperfabric AI to automate fabric provisioning and operations.
Certifications
Highly Desirable:
- NVIDIA-Certified Professional: AI Networking (NCP-AIN)
- NVIDIA-Certified Associate: AI Infrastructure and Operations (NCA-AIIO)
- Cisco Certified Network Professional (CCNP) Data Center
Success Metrics (KPIs)
- Fabric Efficiency: Achieving >95% effective bandwidth efficiency on NCCL-test benchmarks across the provisioned cluster.
- Layer 1 Accuracy: Zero ghost links or unstable connections handed over to the Compute team (all bad cables identified during the burn-in phase).
- Ticket Velocity: Consistently meeting SLAs for network-related support tickets.
