A Beginner’s Guide to Modern Ethernet Fabrics for HPC and AI Environments
vendor-neutral fundamental, packet walks, AI/HPC design context , and troubleshooting
- Part 1 — The Foundations
- Part 2 — VXLAN and EVPN
- Part 3 — Distributed Networking Services
- Part 4 — How The Fabric Works
- Part 5 — HPC and AI Environments
- Part 6 — Operations, Automation, and Troubleshooting
- Part 7 — Design Examples, Glosary, and References← You are here
Design Examples and Use Cases
Small Example Fabric
The following fictional example is deliberately small enough to draw but contains the same conceptual building blocks as a much larger fabric.
Figure 20 – Fictional 2-spine, 4-leaf EVPN/VXLAN fabric.
| Device | VTEP loopback | Illustrative AS |
| Leaf-01 | 10.255.0.11/32 | 65101 |
| Leaf-02 | 10.255.0.12/32 | 65102 |
| Leaf-03 | 10.255.0.13/32 | 65103 |
| Leaf-04 | 10.255.0.14/32 | 65104 |
| Spine-01 | 10.255.255.1/32 | 65000 |
| Spine-02 | 10.255.255.2/32 | 65000 |
| Tenant / VRF | VLAN | Subnet | L2 VNI | L3 VNI | Illustrative RT |
| Tenant-A / VRF-A | 110 | 10.10.110.0/24 | 10110 | 50001 | 65000:50001 |
| Tenant-A / VRF-A | 120 | 10.10.120.0/24 | 10120 | 50001 | 65000:50001 |
| Tenant-B / VRF-B | 210 | 10.20.210.0/24 | 10210 | 50002 | 65000:50002 |
Suppose Host-A1 at 10.10.110.10 attaches to Leaf-01 and Host-A2 at 10.10.120.20 attaches to Leaf-03. Leaf-01 learns Host-A1 locally and advertises a Type 2 MAC/IP route. Leaf-03 does the same for Host-A2. The underlay already knows how to reach both VTEP loopbacks. When Host-A1 sends to Host-A2, Leaf-01 acts as the anycast gateway, routes within VRF-A, encapsulates the routed packet using L3 VNI 50001 toward Leaf-03, and Leaf-03 forwards into local L2 VNI 10120.
Illustrative only
The AS numbering, route-target scheme, VNI values and addressing are examples for learning. Production designs should use a documented allocation model and automated collision checks.
AI Cluster Example
Figure 21 – Conceptual AI factory with separate logical or physical fabrics by traffic class
In this model, the service network uses EVPN/VXLAN because the environment hosts multiple projects, Kubernetes clusters, shared services, and external connectivity. The management network prioritizes simplicity and isolation. The storage network is engineered for sustained throughput and checkpoint bursts. The GPU compute network is optimized for low-latency RDMA and collective traffic, using a routed RoCEv2 Clos or InfiniBand depending on the platform.
A converged Ethernet architecture could combine storage and compute, or even service traffic, on the same physical fabric. That can reduce cabling and switch count, but the architecture must then prove that QoS, queue isolation, congestion control, oversubscription, and failure behaviour protect the compute workload under worst-case mixed traffic.
Common Use Cases
| Use case | Why EVPN/VXLAN can be useful |
| Enterprise private cloud | Creates repeatable tenant VRFs and segments across a routed leaf-spine fabric. |
| Kubernetes platforms | Provides infrastructure segmentation and external/service integration; must be coordinated with the CNI to avoid needless overlay-on-overlay complexity. |
| Multi-tenant data centres | Large VNI space, overlapping address support through VRFs, and policy-driven route import/export. |
| AI training clusters | Useful for service, tenant and management networks; may also be used in compute Ethernet designs when supported and justified. |
| AI inference clusters | Segmentation, scale-out service networking and east-west routing can benefit from distributed gateways. |
| HPC clusters | Useful around general Ethernet services; dedicated compute fabric may instead be plain routed Ethernet/RoCE or InfiniBand. |
| Storage networks | Can segment tenants and provide routed reachability; performance requirements determine whether overlaying storage traffic is appropriate. |
| Bare-metal cloud | Allows logical tenant networks to span physical racks without extending the physical Layer-2 topology. |
| Service-provider data centres | EVPN provides standards-based multi-tenant service constructs and multihoming. |
| Workload mobility | Control-plane endpoint movement can update remote VTEPs more cleanly than large flood-and-learn domains. |
| Data-centre interconnect | EVPN can extend selected services between sites, but DCI introduces failure-domain, latency and policy concerns that require dedicated design. |
Design Trade Offs
Benefits
- Scale-out topology: leaf-spine grows through repeated devices and parallel paths.
- Deterministic routed underlay: Layer-3 links reduce dependence on large spanning-tree domains.
- Active bandwidth: ECMP uses redundant links instead of blocking them.
- Logical scale: VNIs and VRFs provide far more segmentation flexibility than VLAN-only designs.
- Distributed routing: anycast gateways keep first-hop routing close to workloads.
- Control-plane learning: EVPN can distribute endpoint and prefix reachability rather than relying only on flooding.
- Mobility and multihoming: EVPN includes mechanisms for endpoint movement and shared Ethernet segments.
- Automation-friendly repetition: Clos fabrics map naturally to templates, inventories, generated addressing and validation.
- Broad ecosystem: EVPN/VXLAN is implemented across major network operating systems and merchant-silicon platforms.
The benefits reinforce each other. A routed underlay is easier to scale because every link can be active. VXLAN then restores logical flexibility without recreating one giant physical Layer-2 network. EVPN makes the logical layer more explicit and controllable. Automation keeps the resulting state consistent.
Costs and Trade-Offs
| Benefit area | Associated cost or risk |
| More logical segmentation | More state: VNIs, VRFs, route targets and policy must be managed correctly |
| Distributed control plane | Operators need BGP/EVPN troubleshooting skills |
| VXLAN overlay | Encapsulation overhead and MTU planning |
| Large control plane | MAC/IP and prefix scale can stress hardware/software limits |
| Multivendor standards | Interoperability still depends on feature details and implementation maturity |
| All-active paths | Hashing may be uneven for elephant flows |
| Converged AI Ethernet | QoS, congestion, telemetry and failure isolation become more demanding |
| Automation | Bad intent can be deployed consistently and quickly if validation is weak |
EVPN/VXLAN also creates more places where state can disagree: local VLAN-to-VNI mapping, remote EVPN routes, VRF import policy, VTEP reachability, tunnel programming, and hardware forwarding entries. The answer is not to avoid abstraction automatically; it is to make the abstraction observable and machine-validated.
When You Might Not Need EVPN/VXLAN
The best architecture is the simplest design that satisfies the actual requirements. A small cluster with one routed subnet per rack may need only a Layer-3 Clos. A dedicated GPU compute network may benefit more from simple routed RoCEv2 and tuned congestion control than from tenant overlays. Traditional HPC may choose InfiniBand because the ecosystem, management, and performance model fit the workload better.
- Use a plain Layer-3 Clos when workloads can be routed directly and there is no meaningful requirement to stretch logical Layer-2 networks.
- Use routed host connectivity when servers or DPUs can participate directly in the routing model and operational tooling supports it.
- Use dedicated RoCEv2 IP fabrics when RDMA performance is the primary requirement and overlay services provide little value on that path.
- Use InfiniBand when its fabric semantics, congestion control, ecosystem and HPC/AI integration better match the deployment.
- Avoid EVPN/VXLAN in very small networks when the operational cost exceeds the segmentation or mobility benefit.
Architecture principle
Do not choose EVPN/VXLAN because it is fashionable. Choose it when the overlay control-plane and segmentation benefits solve real requirements more cleanly than simpler alternatives.
Glossary
| Term | Beginner-friendly definition |
| ARP | Address Resolution Protocol. IPv4 mechanism used to discover the link-layer address associated with a local IPv4 address. |
| BGP | Border Gateway Protocol. A path-vector routing protocol used on the Internet and inside many large data centres. |
| Clos | A multistage switching topology that provides many parallel paths. Leaf-spine is a common two-tier data-centre form. |
| DF | Designated Forwarder. In EVPN multihoming, the selected PE/VTEP responsible for specific forwarding duties toward an Ethernet segment. |
| ECMP | Equal-Cost Multipath. Installing and using multiple next hops with equal routing cost. |
| ECN | Explicit Congestion Notification. IP mechanism that can mark congestion rather than relying only on packet loss. |
| ESI | Ethernet Segment Identifier. Identifier for an Ethernet segment in EVPN multihoming. |
| Ethernet Segment | A set of Ethernet links attaching a customer/device/network to one or more EVPN PEs/VTEPs. |
| EVPN | Ethernet VPN. A BGP-based control-plane model for distributing Ethernet and IP service reachability. |
| IRB | Integrated Routing and Bridging. Combining Layer-2 bridging and Layer-3 routing within an EVPN service. |
| L2 VNI | A VXLAN VNI representing a Layer-2 overlay/bridge domain. |
| L3 VNI | A VXLAN VNI representing a routed VRF context, commonly used in symmetric IRB. |
| Leaf | Access/fabric switch connecting endpoints and every spine in a leaf-spine topology. |
| MP-BGP | Multiprotocol BGP. BGP extensions that carry multiple address families, including EVPN. |
| ND | Neighbor Discovery. IPv6 mechanisms including neighbor address resolution and router discovery. |
| Overlay | Logical network built on top of an underlying transport network. |
| PFC | Priority Flow Control. IEEE 802.1Qbb mechanism that can pause selected Ethernet priorities on a link. |
| RD | Route Distinguisher. Value used to make otherwise identical VPN routes unique in BGP. |
| RDMA | Remote Direct Memory Access. Direct memory-to-memory data transfer semantics with low CPU involvement. |
| RoCEv2 | RDMA over Converged Ethernet version 2. Carries RDMA transport over UDP/IP and Ethernet. |
| Route Target | BGP extended community commonly used to control import/export membership of VPN/EVPN routes. |
| Spine | Fabric switch connecting to every leaf in a leaf-spine topology. |
| Underlay | Physical/routed transport network that provides connectivity between overlay endpoints. |
| VLAN | Virtual LAN. IEEE 802.1Q Layer-2 segmentation construct identified by a VLAN ID. |
| VNI | VXLAN Network Identifier. 24-bit identifier selecting a VXLAN overlay network. |
| VRF | Virtual Routing and Forwarding instance. Separate Layer-3 routing table used for isolation. |
| VTEP | VXLAN Tunnel Endpoint. Function that encapsulates and decapsulates VXLAN traffic. |
| VXLAN | Virtual eXtensible LAN. UDP/IP overlay encapsulation for carrying Layer-2 network segments across a Layer-3 network. |
References and Further Reading
- RFC 7348 — Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks. https://www.rfc-editor.org/info/rfc7348
- RFC 7432 — BGP MPLS-Based Ethernet VPN. https://www.rfc-editor.org/info/rfc7432
- RFC 7938 — Use of BGP for Routing in Large-Scale Data Centers. https://www.rfc-editor.org/info/rfc7938
- RFC 8365 — A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN). https://www.rfc-editor.org/info/rfc8365
- RFC 9135 — Integrated Routing and Bridging in Ethernet VPN (EVPN). https://www.rfc-editor.org/info/rfc9135
- RFC 9136 — IP Prefix Advertisement in Ethernet VPN (EVPN). https://www.rfc-editor.org/info/rfc9136
- RFC 9161 — Operational Aspects of Proxy ARP/ND in Ethernet Virtual Private Networks. https://www.rfc-editor.org/info/rfc9161
- RFC 8584 — Framework for Ethernet VPN Designated Forwarder Election Extensibility. https://www.rfc-editor.org/info/rfc8584
- RFC 9746 — BGP EVPN Multihoming Extensions for Split-Horizon Filtering. https://www.rfc-editor.org/info/rfc9746
- IEEE 802.1Q — Bridges and Bridged Networks (VLAN and bridging family of standards). https://standards.ieee.org/standard/802_1Q-2022.html
- IEEE 802.1Qbb — Priority-based Flow Control amendment / incorporated into later 802.1Q revisions. https://standards.ieee.org/standard/802_1Qbb-2011.html
- Red Hat Enterprise Linux — Configuring InfiniBand and RDMA Networks — RoCE overview. https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/configuring_infiniband_and_rdma_networks/configuring-roce_configuring-infiniband-and-rdma-networks
- NVIDIA DOCA — RDMA over Converged Ethernet (RoCE), including RoCEv1/v2 packet stack and UDP/IP behaviour. https://docs.nvidia.com/doca/sdk/rdma-over-converged-ethernet/index.html
- NVIDIA Enterprise Reference Architecture — HGX/NVL72 AI Factory network logical architecture, leaf-spine and rail-optimized Ethernet examples. https://docs.nvidia.com/enterprise-reference-architectures/hgx-ai-factory/latest/network-logical-architecture.html