A Beginner’s Guide to NVIDIA Spectrum and Spectrum-X Networking

Spectrum vs ordinary switching, fabric thinking, history, and AI motivation

1. Part 1 — What NVIDIA Spectrum Is
2. Part 2 — Why NVIDIA Spectrum
3. Part 3 — AI Factory Fabric Architecture
4. Part 4 — RDMA and ROCE
5. Part 5 — NOS, IaC, and Operations
6. Part 6 — HA, Physical Layer Design, and Scaling ← You are here
7. Part 7 — Multi-Cloud, Security, and Lifecycle Management
8. Part 8 — Design Scenarios, Glosary, and References

High Availability and Resilience

Clos fabrics are designed around the expectation that links and switches will fail. Resilience comes from parallel paths, fast control-plane convergence, sufficient spare capacity, and small failure domains. The goal is not “nothing ever fails”; the goal is “one failure does not become an outage or severe performance collapse.”

One spine fails

Figure 25. Spine failure

One leaf or rack fails

Figure 26. Leaf/rack failure domain

Design techniques

• Multiple spines and multiple uplinks per leaf.
• ECMP so traffic automatically uses surviving routes.
• Dual-homed servers or multi-rail NIC layouts where the workload needs host-link resilience.
• BFD for fast failure detection where appropriate.
• MLAG for active-active Layer-2 attachment when L2 semantics are necessary.
• Layer-3 multipathing when endpoints and applications can use independent routed interfaces.
• Maintenance mode, graceful routing behaviour, and staged upgrades to drain traffic before work.
• Capacity headroom so losing a spine does not push every surviving link into persistent congestion.

Resilience is capacity-aware

A topology can remain logically connected after a failure yet become practically unusable because the surviving links are oversubscribed. Model degraded-state bandwidth, not just reachability.

Source note: See references [24], [29].

Physical Layer Design

At 400G, 800G, and beyond, the physical layer is not a footnote. Link reach, fiber type, connector density, transceiver thermal load, breakout mapping, rack placement, and cable serviceability can determine whether the logical design is deployable.

Medium	Typical use	Strengths	Watch-outs
DAC	Short in-rack or adjacent-rack links	Low cost, low power, low latency	Short reach, thick/heavy at high speeds
AOC	Short-to-medium reach	Simpler than separate optics/fiber	Less field-repair flexibility
Multimode optics	Data-hall short reach	Familiar structured cabling	Reach/speed combinations vary; fiber plant limits
Single-mode optics	Longer data-hall/campus/DCI reach	High reach and future flexibility	Higher optical cost; cleaning/inspection discipline
Co-packaged optics	Emerging very-high-density systems	Potential power/reliability gains	Different service model and platform-specific cabling

Speed and breakout

Modern Spectrum systems span 100G, 200G, 400G, and 800G-class interfaces, with newer generations moving beyond that at the silicon lane level. A single high-speed front-panel port can often be broken into multiple lower-speed logical ports, but allowed breakout modes depend on switch model, port, optical module, and software release.

Planning principles

• Choose topology and rack layout together with the optical budget.
• Keep a lane/port map that survives breakouts and future recabling.
• Model transceiver and switch power as part of rack thermal design.
• Validate optics against the switch qualification/support list.
• Plan cleaning, inspection, spares, and replacement procedures for optical links.
• Use cable management that allows a failed switch or optic to be replaced without disturbing unrelated links.

Source note: See references [2], [30].

Scale and Capacity Planning

Radix and port arithmetic

Radix is the number of switch ports available for the topology. A leaf uses some ports facing hosts and some facing spines. If a leaf has 32 host-facing 400G ports and 32 400G uplinks, it has 12.8 Tb/s down and 12.8 Tb/s up: a nominal 1:1 non-blocking ratio. If it has 48 downlinks and 16 uplinks at the same speed, the ratio is 3:1 oversubscribed.

Figure 27. Non-blocking versus oversubscribed leaf

Bisection bandwidth

Bisection bandwidth asks how much traffic the fabric can carry when endpoints on one side communicate with endpoints on the other. It is more meaningful for distributed workloads than the sum of all switch port speeds because it reflects how much cross-fabric traffic the topology can sustain.

Worked example: 256 GPU servers

Assume 256 GPU servers, each with one 400G AI-network interface for a simplified example. Use eight leaf switches, 32 servers per leaf. Each leaf therefore has 12.8 Tb/s of host-facing bandwidth. If each leaf also has eight 800G uplinks to eight spines, uplink capacity is 6.4 Tb/s, which creates 2:1 oversubscription at each leaf. That may be acceptable for inference or mixed workloads, but a synchronized training environment may require closer to 1:1 depending on job placement and communication pattern.

If the design must remain at or near 1:1 after one spine failure, the normal-state fabric needs additional headroom. Capacity planning should therefore include failure cases, not only steady state.

Growth planning

• Reserve ports for the next server/rack wave.
• Reserve spine capacity before leaf count reaches the topology limit.
• Decide whether growth means larger switches, more planes, an extra Clos tier, or another pod.
• Budget optics and fiber pathways for future uplinks early; they are difficult to retrofit under dense production cabling.
• Model storage and checkpoint traffic alongside GPU collectives because the peaks can coincide.

Deployment Patterns

Pattern	Typical topology	Spectrum relevance	Main concern
Traditional enterprise DC	Leaf-spine or collapsed fabric	Spectrum for high-performance open Ethernet	Avoid overengineering RoCE/AI features that are not needed
Private cloud	L3 underlay + EVPN/VXLAN	Strong fit for Cumulus/SONiC, automation, multi-tenancy	Operational source of truth and overlay complexity
Kubernetes platform	Routed fabric; optional EVPN; separate high-performance network	ConnectX/BlueField, Network Operator, SR-IOV/RDMA	Pod networking and AI data path are different problems
AI training cluster	Near-non-blocking two-tier or multiplane	Spectrum-X core use case	Congestion, rail mapping, failure headroom
AI inference cluster	High-bandwidth fabric, sometimes more oversubscription	Spectrum-X where distributed inference is network-intensive	Balance cost vs latency and model parallelism needs
HPC over Ethernet	L3 RoCE fabric	Spectrum + RoCE/Spectrum-X	MPI/RDMA tuning and operational discipline
Storage-heavy environment	Redundant L2/L3 storage fabric	Spectrum ESF designs, RoCE/NVMe-oF as applicable	Loss/retransmit sensitivity and failure recovery
Multi-tenant DC	EVPN/VXLAN + VRFs	Open Ethernet and policy scale	Isolation, noisy-neighbour control, automation
Multi-pod AI factory	Pods + inter-pod tier or Spectrum-X Multiplane/XGS	High-scale Spectrum-X direction	Topology/failure domains and cross-pod performance
Brownfield migration	New leaf-spine alongside legacy core	Spectrum integrates through Ethernet/IP	Migration sequencing and asymmetric legacy constraints

Migration principle

For brownfield environments, build the new fabric as a well-defined routing domain and connect it to legacy networks at borders. Avoid importing historical Layer-2 dependencies into the new design unless there is a clear application requirement. Migrate workloads or racks in controlled groups and observe traffic before removing old paths.