A Beginner’s Guide to Modern Ethernet Fabrics for HPC and AI Environments
vendor-neutral fundamental, packet walks, AI/HPC design context , and troubleshooting
- Part 1 — The Foundations
- Part 2 — VXLAN and EVPN ← You are here
- Part 3 — Distributed Networking Services
- Part 4 — How The Fabric Works
- Part 5 — HPC and AI Environments
- Part 6 — Operations, Automation, and Troubleshooting
- Part 7 — Design Examples, Glosary, and References
VXLLAN – The Data Plane Overlay
Why VXLAN Exists
A classic VLAN tag contains a 12-bit VLAN identifier. Because values 0 and 4095 are reserved, there are 4094 usable VLAN IDs in the normal model. That is enough for many enterprise networks but restrictive for large multi-tenant environments, especially when different tenants may reuse the same logical addressing or when logical networks must span a routed physical fabric.
VXLAN creates an overlay segment identified by a 24-bit VXLAN Network Identifier (VNI), allowing roughly 16 million logical segments. An ingress VTEP takes the original frame, adds a VXLAN header plus UDP/IP and outer Ethernet headers, and sends the resulting packet across the IP underlay to an egress VTEP.
The physical fabric therefore forwards ordinary IP packets. It does not need an STP-controlled Layer-2 path between every pair of workloads. VXLAN preserves the useful logical Ethernet abstraction while allowing the physical network to be routed.
| Property | VLAN | VXLAN VNI |
| Identifier width | 12-bit VLAN ID field | 24-bit VNI field |
| Typical scale | 4094 usable VLAN IDs | About 16 million logical segments |
| Physical scope | Usually tied to switched Layer-2 domain | Can span an IP-routed underlay between VTEPs |
| Encapsulation | 802.1Q tag inside Ethernet frame | Original frame carried inside UDP/IP VXLAN encapsulation |
| Control plane | Not defined by VLAN itself | Not defined by VXLAN itself; EVPN is a common control plane |
What VXLAN does not do
VXLAN defines an encapsulation and tunnel model. It does not by itself provide a complete scalable mechanism for advertising every endpoint’s location. That is the problem EVPN commonly solves.
VTEPs and VXLAN Encapsulation
A VXLAN Tunnel Endpoint (VTEP) is the function that adds or removes VXLAN encapsulation. In hardware EVPN/VXLAN fabrics, the leaf switch commonly acts as the VTEP. A hypervisor, DPU, or software switch can also be a VTEP in other architectures.
Figure 6 – Conceptual VXLAN frame. Extra tags or IPv6 change the exact overhead.
The outer Ethernet header changes hop by hop as the packet crosses the physical fabric, just like other routed traffic. The outer IP source and destination identify the tunnel endpoints. The VNI selects the logical overlay context. Inside all of that is the original frame the workload expected to send.
VXLAN adds overhead—commonly about 50 bytes with an outer IPv4 header and no extra outer VLAN tag, with more overhead for IPv6 or additional headers. That means the physical fabric normally needs a larger MTU than the tenant network. If a 1500-byte inner packet must cross a VXLAN fabric, designing the underlay for only a 1500-byte MTU is an invitation to fragmentation or drops.
Operational warning
Validate MTU end to end, including server NICs, leaf interfaces, spine links, port channels, DCI links, and any security or service devices in the path. Overlay MTU problems often masquerade as random application failures.
VLANs, VNIs, L2 VNIs and L3 VNIs
A VLAN and a VNI are not the same object. A VLAN is a local Layer-2 segmentation mechanism on an Ethernet device. A VNI identifies an overlay network carried across VXLAN. Many designs map a local VLAN to a Layer-2 VNI at a VTEP, but the mapping is an implementation choice and does not make the identifiers interchangeable.
An L2 VNI usually represents a bridge domain or subnet segment. An L3 VNI usually represents the routed overlay context associated with a VRF in symmetric IRB designs. A VRF (Virtual Routing and Forwarding instance) is a separate Layer-3 routing table. This allows Tenant-A and Tenant-B to use overlapping IP space without sharing routes.
| Object | Layer | Think of it as | Example |
| VLAN | Layer 2, local | Local switch segmentation label | VLAN 110 |
| L2 VNI | Overlay Layer 2 | Overlay bridge domain identifier | VNI 10110 |
| VRF | Layer 3 | Separate tenant routing table | VRF-AI-Training |
| L3 VNI | Overlay Layer 3 | Overlay routed context for a VRF | VNI 50001 |
| Subnet | Layer 3 addressing | IP network attached to a bridge/gateway | 10.10.110.0/24 |
Figure 7 – One VRF can contain several IP subnets and their corresponding L2 VNIs, plus an L3 VNI for symmetric IRB.
EVPN – The Control Plane
Why VXLAN Alone Is Not Enough
A VXLAN tunnel can carry frames, but the ingress VTEP still needs to know where to send them. Early “flood-and-learn” approaches discover remote locations partly through data-plane flooding: unknown unicast, broadcast, and multicast (BUM) traffic is replicated so that VTEPs can learn endpoint locations from observed frames.
That can work, but the inefficiency becomes increasingly visible as the number of segments and endpoints grows. ARP requests, IPv6 Neighbor Discovery, unknown destinations, and endpoint mobility all create control information that is being inferred indirectly from data traffic. EVPN moves much of that knowledge into an explicit control plane.
With EVPN, a VTEP can advertise “this MAC address and IP address are reachable through me” using MP-BGP. Other VTEPs import the route if policy allows and can then send traffic to the correct remote VTEP without first discovering it through flooding.
What to remember
EVPN reduces dependence on data-plane flood-and-learn. It does not mean all broadcast, unknown-unicast, and multicast traffic disappears in every design.
EVPN Fundamentals
EVPN defines BGP route formats and procedures for Ethernet VPN services. In a VXLAN data centre, EVPN routes can carry MAC addresses, IP addresses, IP prefixes, VTEP-related information, and Ethernet-segment information. MP-BGP distributes those routes between participating network devices.
A Route Distinguisher (RD) makes otherwise identical routes unique in BGP. A Route Target (RT) is a BGP extended community used to control which EVPN routes are imported into or exported from a given service. Beginners often mix them up: the RD is primarily about uniqueness; the RT is primarily about policy and membership.
Figure 8 – A simplified control-plane sequence for endpoint learning.
The exact peer topology varies. Some fabrics use spines as route reflectors for the EVPN overlay. Others peer differently. The important concept is that the EVPN route is carried through BGP independently of the VXLAN data packet that will later use that information.
A little deeper
EVPN was originally specified with MPLS data-plane procedures in RFC 7432. RFC 8365 describes EVPN as a control plane for network virtualization overlays such as VXLAN. The EVPN information model is therefore broader than “VXLAN control plane,” even though EVPN/VXLAN is the dominant data-centre pairing.
EVPN Route Types for Beginners
EVPN defines several route types. You do not need to memorize every field to understand a fabric, but four types appear repeatedly in data-centre operations.
| Route type | Name | What it advertises | Why you care |
| Type 1 | Ethernet Auto-Discovery (A-D) | Ethernet-segment/service information | Important for multihoming, aliasing and fast withdrawal |
| Type 2 | MAC/IP Advertisement | MAC address and optionally associated IP information | Core endpoint reachability; often what operators inspect first |
| Type 3 | Inclusive Multicast Ethernet Tag (IMET) | Membership/tunnel information used for BUM handling | Helps build replication state for a broadcast domain |
| Type 5 | IP Prefix Route | IP prefixes rather than individual MAC entries | Useful for routed connectivity and advertising non-host prefixes |
Type 2 is the route most beginners associate with EVPN because it directly answers “where is this endpoint?” Type 3 is part of how a VTEP learns which other VTEPs participate in an EVPN instance for BUM handling. Type 1 becomes important when an Ethernet segment is attached to more than one PE/VTEP. Type 5, standardized in RFC 9136, extends the model so a fabric can advertise IP prefixes efficiently without requiring every destination to be represented as a MAC/IP host route.
Common misconception
An EVPN route type is a BGP control-plane object. It is not a different kind of VXLAN tunnel.