A Beginner’s Guide to Modern Ethernet Fabrics for HPC and AI Environments
vendor-neutral fundamental, packet walks, AI/HPC design context , and troubleshooting
- Part 1 — The Foundations
- Part 2 — VXLAN and EVPN
- Part 3 — Distributed Networking Services ← You are here
- Part 4 — How The Fabric Works
- Part 5 — HPC and AI Environments
- Part 6 — Operations, Automation, and Troubleshooting
- Part 7 — Design Examples, Glosary, and References
Distributed Layer 2 and Layer 3 Services
Anycast Gateway
In an older design, a subnet’s default gateway might live on one pair of aggregation switches. A server attached elsewhere sends inter-subnet traffic toward that centralized gateway, which can create tromboning and makes workload mobility dependent on the gateway’s physical location.
A distributed anycast gateway places the same logical default-gateway IP—and normally the same virtual gateway MAC—on multiple VTEPs. A host therefore sends traffic to a local leaf for first-hop routing even after moving to another rack. This is data-centre anycast gateway behaviour, not Internet anycast routing.
Figure 9 – Distributed first-hop routing removes unnecessary trips to a centralized gateway.
| Centralized gateway | Distributed anycast gateway |
| Gateway exists at selected aggregation point | Same logical gateway exists on participating leaf VTEPs |
| Inter-subnet traffic may trombone | Ingress leaf can route immediately |
| Mobility may change path to gateway | Gateway stays topologically local to endpoint |
| Fewer gateway instances | More distributed state; requires consistent automation |
Integrated Routing and Bridging (IRB)
Integrated Routing and Bridging (IRB) combines Layer-2 bridging within a subnet with Layer-3 routing between subnets. EVPN/VXLAN fabrics commonly use IRB so workloads can retain Ethernet semantics inside a subnet while routing is distributed across the leaf switches.
In asymmetric IRB, the ingress VTEP routes from the source subnet into the destination subnet’s L2 VNI. The ingress therefore needs knowledge of both source and destination Layer-2 segments. This can be simple for small environments but scales poorly when every VTEP must instantiate many remote L2 VNIs.
In symmetric IRB, the ingress VTEP routes from the source L2 VNI into an L3 VNI associated with the tenant VRF. The egress VTEP then routes from the L3 VNI into the local destination L2 VNI. Each VTEP can focus on the local L2 segments it actually serves while sharing the tenant’s routed overlay.
Figure 10 – Symmetric IRB uses an L3 VNI for the routed portion of the overlay path.
| Characteristic | Asymmetric IRB | Symmetric IRB |
| Routed overlay representation | Destination L2 VNI is used after ingress routing | Tenant L3 VNI carries routed traffic between VTEPs |
| Remote L2 state on ingress | Often requires more remote L2 VNIs | Can keep L2 VNIs localized |
| Scale tendency | Convenient at smaller scale | Common choice for larger fabrics |
| Operational model | Fewer moving concepts initially | Clearer separation of tenant routing and local bridging at scale |
A little deeper
RFC 9135 specifies EVPN procedures for Integrated Routing and Bridging. Vendor syntax and some gateway behaviours vary, so learn the conceptual packet walk first and map it to a platform later.
ARP and Neighbor Discovery Suppression
IPv4 hosts use ARP to discover the MAC address associated with a local IPv4 address. IPv6 uses Neighbor Discovery (ND) for similar neighbor-resolution functions. In a large stretched Layer-2 segment, broadcasting or multicasting every resolution request to every remote VTEP wastes bandwidth and endpoint CPU.
When EVPN carries reliable MAC/IP bindings, a VTEP can answer some ARP or ND requests locally on behalf of remote endpoints. This is generally called ARP/ND suppression or proxy behaviour. It reduces broadcast/multicast replication and improves scale, especially in large multi-tenant environments.
Suppression is not “ARP has been removed.” Local hosts still use ARP or ND. The fabric is simply using control-plane knowledge to avoid sending every query across the overlay. Operational correctness therefore depends on accurate endpoint learning and mobility handling.
Resilience and Multihoming
ECMP and Fabric Resilience
ECMP lets a router install several next hops for the same destination when their routing cost is equal. Leaf-spine designs intentionally create these equal-cost alternatives. Traffic is usually distributed by hashing header fields so packets from one flow follow a stable path while different flows can use different spines.
Figure 11 – A single failed uplink removes one next hop without requiring a Layer-2 topology reconvergence.
Hashing is not the same as perfect balancing. A few very large “elephant” flows can collide on one path even when many paths are available. Modern AI fabrics may therefore use more advanced adaptive-routing or flowlet techniques, but the base IP fabric still depends on correct routing and path diversity.
Why this matters for AI
A fabric can have enormous aggregate bandwidth and still perform badly if synchronized GPU flows concentrate on a small number of links. Topology, hashing entropy, adaptive routing, and congestion control all matter.
EVPN Multihoming
A server, storage system, firewall, or downstream switch may connect to two leaf switches for resilience. EVPN multihoming gives the EVPN control plane a standard way to represent that shared attachment and coordinate forwarding.
Figure 12 – An Ethernet Segment can attach the same endpoint or downstream domain to multiple EVPN PEs/VTEPs.
The Ethernet Segment Identifier (ESI) identifies the shared Ethernet segment. Depending on design, the segment can operate all-active, where both links can forward, or single-active, where only one attachment forwards certain traffic. A Designated Forwarder (DF) election controls specific forwarding responsibilities such as BUM delivery toward the shared segment.
EVPN multihoming also supports ideas such as split-horizon, aliasing, and mass withdrawal. Split-horizon prevents traffic received from an Ethernet segment from being sent back into the same multihomed segment incorrectly. Aliasing can allow remote devices to use multiple attachment points even before every individual MAC is learned from each one. Mass withdrawal accelerates convergence by withdrawing reachability associated with a failed segment rather than waiting for every MAC entry to age independently.
Operational warning
Multihoming is powerful but substantially increases control-plane complexity. Prove the single-homed fabric first, then add Ethernet-segment behaviour and validate failure cases deliberately.