A Beginner’s Guide to Modern Ethernet Fabrics for HPC and AI Environments
vendor-neutral fundamental, packet walks, AI/HPC design context , and troubleshooting
- Part 1 — The Foundations
- Part 2 — VXLAN and EVPN
- Part 3 — Distributed Networking Services
- Part 4 — How The Fabric Works
- Part 5 — HPC and AI Environments
- Part 6 — Operations, Automation, and Troubleshooting ← You are here
- Part 7 — Design Examples, Glosary, and References
Keeping the Lights On
Fabric Automation and Source of Truth
A fabric with hundreds of switches is mostly repeated structure: interfaces, addressing, AS numbers, VNIs, route targets, VRFs, and policy templates. Configuring those by hand turns repetition into risk. Modern operations therefore start with a source of truth that describes intended state and generate device-specific configuration from structured data.
A robust workflow resembles software delivery: change intent in version control, validate syntax and topology, run policy and consistency tests, render configurations, stage changes, observe results, and detect drift. Network CI/CD is not about pretending switches are stateless software; it is about applying software-engineering discipline to infrastructure change.
Figure 18 – A repeatable automation pipeline reduces per-device variation and makes rollback criteria explicit.
Operational warning
Route targets, VNIs, anycast gateway values, MTU, and BGP policies are excellent candidates for machine validation because one inconsistent value can create failures that are difficult to see from a single switch.
Observability
A healthy fabric needs visibility at several layers. “BGP is up” is not sufficient. Operators should be able to see physical errors, underlay routes, VTEP reachability, EVPN routes, MAC/IP bindings, VXLAN state, VRFs, queue occupancy, ECN marks, PFC events where used, latency, loss, and flow distribution.
| Healthy control plane might show | Healthy data plane should also show |
| BGP neighbors Established | Traffic actually forwards at expected rate |
| VTEP loopbacks reachable | No MTU-related loss or fragmentation |
| Expected EVPN Type 2/5 routes installed | Correct encapsulation and decapsulation counters |
| Correct RT import/export | Correct VRF/VNI forwarding context |
| Ethernet-segment state converged | No duplicate delivery or split-horizon error |
| ARP/ND bindings present | Hosts resolve peers and gateways correctly |
| All expected ECMP next hops installed | Flows distribute acceptably across links |
For AI/RDMA fabrics, add queue and congestion telemetry to the normal routing view. ECN marking rate, congestion notification behaviour, PFC pause counters, buffer occupancy, retransmission indicators, and per-link utilization can reveal performance faults long before a link goes administratively down.
Troubleshooting Method
Troubleshoot from the simplest layer upward. Resist the temptation to start with the most exotic EVPN command because the symptom mentions VXLAN. A repeatable sequence narrows the fault domain quickly.
- Is the physical interface up, error-free, and at the expected speed?
- Is the underlay routing healthy and are ECMP next hops present?
- Can the VTEP loopbacks reach each other with the required MTU?
- Is the MP-BGP EVPN session established?
- Is the expected EVPN route present and valid?
- Was the route imported into the correct VRF/VNI according to route-target policy?
- Is the endpoint MAC/IP or IP prefix learned correctly?
- Is VXLAN encapsulation/decapsulation occurring and are counters increasing?
- Are MTU, ACLs, firewall rules, or QoS policies dropping traffic?
- Is ECMP hashing distributing the relevant flows?
- For RoCEv2, are ECN/PFC/congestion-control signals behaving as designed?
- Does the failure reproduce after moving traffic to a different path or endpoint?
Figure 19 – A layered troubleshooting decision flow.
Common Beginner Mistakes
| Mistake | Correction |
| “VXLAN is a routing protocol.” | VXLAN is an encapsulation. Routing is performed by the underlay and by VRF/IRB forwarding. |
| “EVPN forwards packets.” | EVPN distributes control-plane reachability; the data plane forwards traffic. |
| “BGP is only for the Internet.” | BGP is also widely used as a data-centre routing/control-plane protocol. |
| “A VNI is just a bigger VLAN ID.” | A VNI identifies an overlay network; mapping to local VLANs is a VTEP implementation detail. |
| “If EVPN is up, the network is healthy.” | MTU, ASIC programming, ACLs, queues, host state and hashing can still break the data plane. |
| “Overlays improve performance.” | Overlays improve logical flexibility and scale; they add headers and operational state. |
| “EVPN makes Ethernet lossless.” | Lossless/congestion behaviour is governed by Ethernet QoS and transport/congestion-control mechanisms. |
| “RoCEv2 needs EVPN.” | RoCEv2 can run on a plain routed IP fabric. |
| “More Layer 2 is more flexible.” | Stretch only what needs stretching; routed boundaries usually reduce failure domains. |
| “Aggregate bandwidth tells me AI performance.” | Traffic pattern, oversubscription, queueing, entropy and tail behaviour also matter. |