A Beginner’s Guide to NVIDIA Spectrum and Spectrum-X Networking

Spectrum vs ordinary switching, fabric thinking, history, and AI motivation

1. Part 1 — What NVIDIA Spectrum Is
2. Part 2 — Why NVIDIA Spectrum
3. Part 3 — AI Factory Fabric Architecture
4. Part 4 — RDMA and ROCE
5. Part 5 — NOS, IaC, and Operations ← You are here
6. Part 6 — HA, Physical Layer Design, and Scaling
7. Part 7 — Multi-Cloud, Security, and Lifecycle Management
8. Part 8 — Design Scenarios, Glosary, and References

Network Operating Systems

Cumulus Linux

Cumulus Linux is the strategic Linux NOS for NVIDIA Spectrum switching. Current Cumulus 5.x releases are based on Debian and use NVUE for a structured operational interface while retaining normal Linux tools underneath. Routing is implemented through familiar open networking components, and configuration can be automated with APIs and configuration-management systems.

September 2026 version nuance: Cumulus Linux 5.18 exists and supports Spectrum-6 SN6600_LD, but NVIDIA states 5.18.0 is currently qualified only for non-Spectrum-X. The August 2026 Spectrum-X validated solution stack instead lists Cumulus 5.16.6 for Spectrum-4. In production, the validated stack takes precedence over “latest standalone release.”

SONiC

NVIDIA Spectrum hardware also supports SONiC on qualified platforms. SONiC appeals to organizations that standardize on the open-source cloud switching ecosystem and can operate its distributed services model. Qualification differs by switch generation and model, so use the hardware support matrix rather than assume every image runs everywhere.

Onyx

NVIDIA Onyx remains documented and available on selected Spectrum systems. It provides a more traditional network-appliance experience with an industry-style CLI and integrated features such as RoCE configuration and automation. For new AI-scale designs, NVIDIA’s current Spectrum-X positioning emphasizes Cumulus and SONiC, so Onyx is best understood as a relevant operational option for particular platforms and estates rather than the default teaching model for Spectrum-X.

Operational style	Cumulus Linux	SONiC	Onyx
Mental model	Linux server + switch ASIC	Cloud NOS made of containerized/services components	Traditional network appliance
Primary interface	NVUE + Linux + APIs	CLI/APIs/config DB tooling	Traditional CLI/Web/API options
Automation fit	Very strong for Linux/Git/Ansible teams	Strong for cloud-scale engineering teams	Strong, but more appliance-oriented
Spectrum-X relevance	Core/current	Core/current on qualified systems	More platform/estate dependent
Learning curve	Linux networking + routing	SONiC architecture + service model	Familiar for classic network engineers

Automation and Infrastructure as Code

At large scale, the question “what commands do I type?” becomes less important than “what desired state do I want, how do I validate it, and how do I prove production matches it?” Network infrastructure as code applies software-engineering discipline to configuration changes.

Figure 23. Network delivery pipeline

Core practices

• Declarative intent: describe the state you want rather than relying on a long sequence of manual commands.
• Source of truth: keep device roles, links, IP plans, ASNs, VRFs, and expected policies in a structured authoritative repository.
• Git workflow: every change is reviewable, attributable, and reversible.
• CI checks: validate schemas, address plans, BGP relationships, MTU, QoS profiles, forbidden commands, and expected route counts.
• Pre-production simulation: run configurations and automation against DSX Air or physical lab topologies.
• Staged rollout: change a subset of devices or one failure domain before the entire fabric.
• Drift detection: compare intended configuration with running state.
• Rollback: preserve a tested recovery path before deployment.

Tools

Ansible is common for configuration orchestration; Python is useful for data transformation, validation, API workflows, and custom tooling. NVUE provides structured commands and APIs on Cumulus. Larger organizations may add a network source-of-truth platform, CI systems, secret management, and change-ticket integration.

DSX Air

DSX Air is NVIDIA’s cloud-hosted data-centre simulation platform. It can build custom topologies, import topology files, use zero-touch provisioning, and execute automation against simulated devices. It is especially useful for learning and for validating the logic of fabric changes, although it does not replace physical testing for optical behaviour, ASIC scale limits, or end-to-end GPU performance.

Source note: See references [14], [27].

Observability and Operations

A large fabric cannot be operated by staring at interface up/down state. The goal is to understand whether the network is healthy for the application. That means correlating link state, routing, queue depth, ECN, PFC, drops, optical errors, latency, host/NIC state, and application-level symptoms.

Signals to watch

• Physical: link state, BER/FEC counters, optic power/temperature, lane health.
• Forwarding: interface utilization, packet/byte rate, errors, discards, route/ECMP state.
• Queues: queue occupancy, buffer usage, tail drops, ECN marks.
• RoCE: priority mapping, PFC pause frames if enabled, CNP/congestion behaviour, NIC state.
• Control plane: BGP adjacency changes, route churn, BFD events, CPU/memory pressure.
• Fabric: path imbalance, adaptive-routing state, topology changes, What Just Happened events where supported.
• Application: NCCL bandwidth, job completion time, GPU utilization, storage throughput.

NetQ

NetQ correlates network state and events across hosts and switches. Current NetQ releases include Spectrum-specific views and, on supported generations, adaptive-routing, RoCE, queue, and multiplane observability. NetQ 5.3 is current as of September 2026. Scale deployments are particularly relevant for Spectrum-X and mixed Ethernet/NVLink environments.

Streaming telemetry

gNMI and OpenTelemetry-style integrations allow operational data to stream into central systems. Prometheus/Grafana can be useful for dashboards, but avoid converting every counter into a graph. The highest-value telemetry connects a network event to an application impact and a specific failure domain.

Figure 24. “GPU application is slow” troubleshooting decision tree

Source note: See references [10], [21], [28].