A Beginner’s Guide to Modern Ethernet Fabrics for HPC and AI Environments
vendor-neutral fundamental, packet walks, AI/HPC design context , and troubleshooting
- Part 1 — The Foundations ← You are here
- Part 2 — VXLAN and EVPN
- Part 3 — Distributed Networking Services
- Part 4 — How The Fabric Works
- Part 5 — HPC and AI Environments
- Part 6 — Operations, Automation, and Troubleshooting
- Part 7 — Design Examples, Glosary, and References
The one-sentence version
EVPN, VXLAN, and BGP are related but different. VXLAN encapsulates traffic. EVPN advertises reachability. BGP transports routing information. The physical underlay forwards the outer IP packet.
The central mental model: logical services ride over a routed physical fabric.
Overview
Modern data-centre fabrics separate physical IP reachability from logical tenant and workload networks. A leaf-spine underlay provides many equal-cost routed paths. VXLAN carries logical Layer-2 or Layer-3 segments across that underlay. EVPN, carried as an MP-BGP address family, distributes endpoint and network reachability so the fabric can make better forwarding decisions than a flood-and-learn overlay alone.
This guide starts with that mental model and then builds the details gradually: leaf-spine architecture, the BGP underlay, VXLAN encapsulation, EVPN route types, distributed gateways, IRB, multihoming, packet walks, automation, observability, and troubleshooting. The final third focuses on HPC and AI environments, where east-west bandwidth, RDMA, congestion control, multi-rail topology, and failure behaviour change the design priorities.
What Is an EVPN/VXLAN/BGP Fabric?
In plain English, an EVPN/VXLAN/BGP fabric is a network in which the physical switches form a routed IP mesh, while logical networks are carried across that mesh as overlays. BGP is commonly used to build IP reachability and to distribute EVPN information. VXLAN is the encapsulation that carries an original Ethernet frame—or, in some designs, routed tenant traffic—between tunnel endpoints called VTEPs.
The most useful beginner mental model is to picture two networks at the same time. The underlay is the real network of cables, switch ports, IP addresses, and routes. The overlay is the logical view presented to applications and tenants: subnets, bridge domains, VRFs, and VNIs. The overlay depends on the underlay, but the underlay does not need to understand every tenant MAC address or subnet.
Figure 1 – Underlay versus overlay: the overlay is logical; the underlay is the physical forwarding medium.
A second separation matters just as much: control plane versus data plane. The control plane learns and distributes information used to make forwarding decisions. The data plane moves packets and frames using the resulting forwarding tables. EVPN is primarily control plane. VXLAN is primarily data-plane encapsulation. The ASIC in a switch forwards the actual traffic.
Common misconception
EVPN does not encapsulate user packets, and VXLAN does not by itself tell every VTEP where each endpoint lives. EVPN and VXLAN solve complementary problems.
Why Data-Centre Networking Changed
Traditional enterprise data centres often extended Layer 2 through several tiers of switching. Spanning Tree Protocol (STP) prevented loops by blocking redundant paths. This was understandable when most traffic was north-south—client to server or server to external network—but it wastes links and creates awkward failure domains when servers communicate heavily with one another.
Virtualization, private cloud, containers, and large distributed systems increased the number of endpoints and made east-west communication dominant. Operators also wanted small repeatable switches instead of a few very large chassis, and they wanted automation that could configure thousands of similar links predictably. A routed Clos/leaf-spine topology matched these needs because every leaf can reach every other leaf through multiple equal-cost spine paths.
| Traditional tendency | Modern fabric tendency |
| Large Layer-2 domains | Routed Layer-3 underlay with smaller physical failure domains |
| STP blocks redundant links | ECMP actively uses many equal-cost paths |
| Chassis scale-up | Leaf/spine horizontal scale-out |
| VLAN-centric segmentation | VRF/VNI-based logical segmentation |
| Flood-and-learn endpoint discovery | Control-plane distribution with EVPN where applicable |
| Manual per-device configuration | Templates, source of truth, validation and CI/CD |
| Mostly north-south assumptions | Heavy east-west traffic, especially AI/HPC |
AI and HPC amplify the east-west problem. Distributed training may cause hundreds or thousands of accelerators to exchange synchronized data. Scale-out storage and checkpointing can add large flows at the same time. A fabric therefore needs not only connectivity but predictable path count, high bisection bandwidth, clear failure behaviour, and observable congestion.
A Short History
Ethernet first gave data centres a simple shared link-layer model. VLANs, standardized in IEEE 802.1Q, added logical separation inside a switched network. STP and its successors kept those networks loop-free, but at the cost of disabling some redundant paths. As data centres grew, routed Layer 3 boundaries were pushed closer to servers to reduce Layer-2 failure domains and make all links usable.
Clos topologies—often implemented as leaf-spine—provided many short parallel paths using repeated building blocks. BGP, historically associated with the Internet, proved attractive inside very large data centres because it is mature, policy-capable, horizontally scalable, and well suited to simple point-to-point routed adjacencies. RFC 7938 documented a practical eBGP approach for large-scale data-centre routing.
Virtualization then created a different requirement: keep the operational advantages of a routed IP fabric while offering logical segments that can span racks. VXLAN, documented in RFC 7348, added a 24-bit VNI and UDP/IP encapsulation. EVPN, defined first for BGP/MPLS in RFC 7432 and adapted to network virtualization overlays in RFC 8365, supplied a BGP-based control plane for endpoint and service reachability.
Foundation Concepts
Leaf-Spine Architecture
In a leaf-spine fabric, endpoints connect to leaf switches. Every leaf connects to every spine in the same fabric plane. Spines normally do not connect directly to endpoints, and leaves normally do not connect directly to other leaves. The result is a two-stage Clos network with a predictable number of hops between racks.
Figure 2 – A simple two-spine, four-leaf fabric. Every leaf has a path through every spine.
Because the leaf-to-spine links have equal routing cost, the forwarding plane can use Equal-Cost Multipath (ECMP). Instead of blocking redundant links, the network hashes flows across several active next hops. Adding spines can increase aggregate fabric capacity until leaf port limits or other design boundaries are reached.
Oversubscription describes the ratio between potential server-facing bandwidth and uplink bandwidth. A 1:1 design is often called nonblocking at the intended tier because the total leaf downlink capacity is matched by uplink capacity. Many enterprise networks deliberately oversubscribe because workloads rarely transmit at line rate simultaneously. Large distributed AI jobs can make that assumption unsafe, so compute fabrics are often designed with much lower oversubscription.
Figure 3 – East-west traffic has multiple active paths instead of a single STP-selected tree.
What to remember
Leaf-spine is a topology, not an overlay protocol. You can build a perfectly valid leaf-spine network with plain Layer-3 routing and no VXLAN or EVPN at all.
The IP Underlay
The underlay’s job is deliberately simple: provide resilient IP reachability between infrastructure addresses, especially VTEP loopbacks. A common design uses numbered or unnumbered point-to-point Layer-3 links between leaves and spines, plus loopback addresses that remain stable even when an individual physical link changes state.
Any routing protocol that meets the requirements can build this reachability. Some fabrics use OSPF or IS-IS. Many large designs use eBGP because each point-to-point adjacency can be modeled simply, operational boundaries are explicit, and BGP already exists for the EVPN overlay. Using BGP twice does not mean the two functions are the same: the underlay address family carries infrastructure IP prefixes, while the overlay EVPN address family carries EVPN routes.
Figure 4 – Illustrative underlay addressing. The exact scheme varies by implementation.
The underlay generally does not need to know Tenant-A’s MAC addresses, every VM subnet, or every Kubernetes service. It just forwards outer IP packets between VTEPs. Keeping this role narrow makes the physical network easier to reason about and troubleshoot.
BGP for Beginners
Routing protocols let routers exchange reachability. BGP is a path-vector protocol: it advertises prefixes together with attributes that describe how those prefixes can be reached and how paths should be selected. BGP speakers form explicit neighbor relationships, exchange routes, and install selected paths into routing tables.
An Autonomous System (AS) is a routing domain identified by an AS number. eBGP operates between different AS numbers; iBGP operates within the same AS. Data-centre designs often assign AS numbers in ways that make leaf-spine relationships easy to automate—for example one AS per leaf and one common or grouped AS for spines—but many models are valid.
Figure 5 – BGP peers exchange reachability and attributes; they do not forward user traffic merely because a BGP session exists.
MP-BGP (Multiprotocol BGP) extends BGP so it can carry multiple address families. IPv4 unicast is one address family. IPv6 unicast is another. EVPN is another. That is why you will often hear “MP-BGP EVPN”: BGP is the transport and route-selection framework, while EVPN defines the information model carried in that address family.
| BGP role | Typical information | Purpose |
| Underlay BGP | Infrastructure IP prefixes, loopbacks, point-to-point reachability | Make VTEP and fabric addresses reachable |
| MP-BGP EVPN overlay | MAC/IP advertisements, IMET information, IP prefixes, Ethernet-segment information | Distribute logical service and endpoint reachability |
Common misconception
BGP is not “only for the Internet.” The protocol is used inside large data centres because its operational model also fits horizontally scaled Clos fabrics.