A Beginner’s Guide to Modern Ethernet Fabrics for HPC and AI Environments
vendor-neutral fundamental, packet walks, AI/HPC design context , and troubleshooting
- Part 1 — The Foundations
- Part 2 — VXLAN and EVPN
- Part 3 — Distributed Networking Services
- Part 4 — How The Fabric Works ← You are here
- Part 5 — HPC and AI Environments
- Part 6 — Operations, Automation, and Troubleshooting
- Part 7 — Design Examples, Glosary, and References
Let’s Take Some Walks
End-to-End Packet Walks
Packet walks are the fastest way to turn EVPN/VXLAN from a vocabulary exercise into a working mental model. The following examples assume symmetric IRB, hardware leaf VTEPs, and an already converged control plane.
Same leaf, same subnet
1. Host A determines that Host B is in the same IP subnet.
2. Host A resolves Host B’s MAC address using ARP or ND.
3. Both hosts are local to Leaf-01, so Leaf-01 bridges the frame locally.
4. No VXLAN encapsulation is required because the frame never leaves the local VTEP.
Figure 13 – Same leaf, same subnet packet walk
Different leaves, same subnet
1. Host A determines that Host B is in the same subnet and sends an Ethernet frame toward Host B’s MAC.
2. Leaf-01 already has an EVPN Type 2 route showing Host B behind Leaf-02’s VTEP.
3. Leaf-01 maps the local bridge domain to L2 VNI 10110 and encapsulates the original frame in VXLAN.
4. The underlay routes the outer IP packet from Leaf-01’s VTEP address to Leaf-02’s VTEP address using ECMP.
5. Leaf-02 removes the VXLAN encapsulation and bridges the original frame to Host B.
Figure 14 – Different leaves, same subnet packet walk
Different leaves, different subnets
1. Host A sees that Host B is outside its subnet and sends the packet to the anycast default gateway MAC on Leaf-01.
2. Leaf-01 performs Layer-3 routing in Tenant-A’s VRF.
3. The routed packet is associated with Tenant-A’s L3 VNI 50001 for the overlay path.
4. Leaf-01 VXLAN-encapsulates toward Leaf-02 using information learned through EVPN.
5. Leaf-02 decapsulates, performs the egress routing/adjacency lookup, maps into destination L2 VNI 10120, and forwards to Host B.
Figure 15 – Different leaves, different subnets packet walk
Inter-VRF or inter-tenant traffic
VRFs are separate routing domains by design. Traffic does not cross between them merely because both exist on the same switches. If the architecture permits shared services or controlled inter-tenant communication, routes must be deliberately leaked or the traffic must pass through an appropriate firewall/service insertion point. The exact mechanism is a policy decision, not an automatic property of EVPN.
Traffic entering or leaving the fabric
North-south traffic usually reaches a border leaf or service leaf that connects the EVPN/VXLAN fabric to external routing domains, firewalls, WANs, or Internet edges. EVPN Type 5 IP prefix routes can be useful for distributing external or summarized prefixes into tenant VRFs, depending on design. The border still needs conventional routing policy and failure handling.
A dual-homed endpoint
For an all-active Ethernet segment, remote VTEPs may have multiple valid next-hop attachment points toward the dual-homed system. EVPN Ethernet-segment routes coordinate that state. If one attachment fails, control-plane withdrawal can remove the failed path rapidly while the other attachment remains usable.
Link or leaf failure
A spine-link failure is primarily an underlay event: the affected ECMP next hop disappears, but the overlay route may remain valid because the destination VTEP still exists. A leaf failure is different: the VTEP and endpoints behind it become unreachable, so both underlay reachability and EVPN endpoint routes must converge. This is why separating failure domains mentally is so useful when troubleshooting.
Control Plane Versus Data Plane
| Function | Typical technology or construct |
| Physical IP reachability | Underlay routing, commonly BGP/OSPF/IS-IS |
| Multipath forwarding | ECMP |
| Overlay encapsulation | VXLAN |
| Endpoint and service distribution | EVPN over MP-BGP |
| Tenant separation | VRFs + VNIs + route-target policy |
| Distributed first-hop routing | Anycast gateway + IRB |
| Actual forwarding | Switch/NIC data plane using programmed forwarding tables |
When a packet fails, ask which layer owns the failed function. If VTEP loopbacks cannot reach each other, troubleshooting EVPN MAC routes first is premature. If the underlay is healthy but a Type 2 route is missing, the problem is probably control-plane learning or policy. If the route exists but large packets fail, investigate data-plane MTU, queues, ACLs, or hardware programming.
What to remember
The underlay must work before the overlay can work. The control plane can be healthy while the data plane is broken, and the reverse can also happen.