A Beginner’s Guide to Modern Ethernet Fabrics for HPC and AI Environments
vendor-neutral fundamental, packet walks, AI/HPC design context , and troubleshooting
- Part 1 — The Foundations
- Part 2 — VXLAN and EVPN
- Part 3 — Distributed Networking Services
- Part 4 — How The Fabric Works ← You are here
- Part 5 — HPC and AI Environments
- Part 6 — Operations, Automation, and Troubleshooting
- Part 7 — Design Examples, Glosary, and References
Let’s Take Some Walks
End-to-End Packet Walks
Packet walks are the fastest way to turn EVPN/VXLAN from a vocabulary exercise into a working mental model. The following examples assume symmetric IRB, hardware leaf VTEPs, and an already converged control plane.
Same leaf, same subnet
1. Host A determines that Host B is in the same IP subnet.
2. Host A resolves Host B’s MAC address using ARP or ND.
3. Both hosts are local to Leaf-01, so Leaf-01 bridges the frame locally.
4. No VXLAN encapsulation is required because the frame never leaves the local VTEP.
Figure 13 – Same leaf, same subnet packet walk
Different leaves, same subnet
1. Host A determines that Host B is in the same subnet and sends an Ethernet frame toward Host B’s MAC.
2. Leaf-01 already has an EVPN Type 2 route showing Host B behind Leaf-02’s VTEP.
3. Leaf-01 maps the local bridge domain to L2 VNI 10110 and encapsulates the original frame in VXLAN.
4. The underlay routes the outer IP packet from Leaf-01’s VTEP address to Leaf-02’s VTEP address using ECMP.
5. Leaf-02 removes the VXLAN encapsulation and bridges the original frame to Host B.
Figure 14 – Different leaves, same subnet packet walk
Different leaves, different subnets
- Host A sends the packet to its default gateway.
Host A at 10.10.110.10/24 determines that Host B at 10.10.120.20/24 is in a different subnet, so it sends the Ethernet frame to the anycast default-gateway MAC on Leaf-01. - Leaf-01 performs Layer-3 routing in Tenant-A’s VRF.
Leaf-01 receives the frame, removes the local Layer-2 forwarding context, and performs an IP route lookup for 10.10.120.20 inside Tenant-A’s VRF. The resulting route identifies Leaf-02 as the remote egress VTEP. - Leaf-01 encapsulates the routed packet using L3 VNI 50001.
Because this is a symmetric IRB flow between different subnets, Leaf-01 places the routed packet into Tenant-A’s L3 VNI 50001 and VXLAN-encapsulates it toward Leaf-02. The destination VTEP information has been learned through EVPN. - The IP underlay carries the VXLAN packet to Leaf-02.
The physical leaf-spine fabric forwards the packet using only the outer IP header, whose source and destination are the Leaf-01 and Leaf-02 VTEP addresses. ECMP may select any valid equal-cost path through the underlay. - Leaf-02 decapsulates, performs the egress lookup, and forwards to Host B.
Leaf-02 removes the VXLAN/L3 VNI 50001 encapsulation, performs the egress routing and adjacency lookup in Tenant-A’s VRF, and maps the destination into L2 VNI 10120 / subnet 10.10.120.0/24. Because Host B is directly attached to Leaf-02, Leaf-02 sends a normal Ethernet frame to 10.10.120.20—there is no second VXLAN encapsulation on the final hop.
Figure 15 – Different leaves, different subnets packet walk
Inter-VRF or inter-tenant traffic
VRFs are separate routing domains by design. Traffic does not cross between them merely because both exist on the same switches. If the architecture permits shared services or controlled inter-tenant communication, routes must be deliberately leaked or the traffic must pass through an appropriate firewall/service insertion point. The exact mechanism is a policy decision, not an automatic property of EVPN.
Traffic entering or leaving the fabric
North-south traffic usually reaches a border leaf or service leaf that connects the EVPN/VXLAN fabric to external routing domains, firewalls, WANs, or Internet edges. EVPN Type 5 IP prefix routes can be useful for distributing external or summarized prefixes into tenant VRFs, depending on design. The border still needs conventional routing policy and failure handling.
A dual-homed endpoint
For an all-active Ethernet segment, remote VTEPs may have multiple valid next-hop attachment points toward the dual-homed system. EVPN Ethernet-segment routes coordinate that state. If one attachment fails, control-plane withdrawal can remove the failed path rapidly while the other attachment remains usable.
Link or leaf failure
A spine-link failure is primarily an underlay event: the affected ECMP next hop disappears, but the overlay route may remain valid because the destination VTEP still exists. A leaf failure is different: the VTEP and endpoints behind it become unreachable, so both underlay reachability and EVPN endpoint routes must converge. This is why separating failure domains mentally is so useful when troubleshooting.
Control Plane Versus Data Plane
| Function | Typical technology or construct |
| Physical IP reachability | Underlay routing, commonly BGP/OSPF/IS-IS |
| Multipath forwarding | ECMP |
| Overlay encapsulation | VXLAN |
| Endpoint and service distribution | EVPN over MP-BGP |
| Tenant separation | VRFs + VNIs + route-target policy |
| Distributed first-hop routing | Anycast gateway + IRB |
| Actual forwarding | Switch/NIC data plane using programmed forwarding tables |
When a packet fails, ask which layer owns the failed function. If VTEP loopbacks cannot reach each other, troubleshooting EVPN MAC routes first is premature. If the underlay is healthy but a Type 2 route is missing, the problem is probably control-plane learning or policy. If the route exists but large packets fail, investigate data-plane MTU, queues, ACLs, or hardware programming.
What to remember
The underlay must work before the overlay can work. The control plane can be healthy while the data plane is broken, and the reverse can also happen.