-
A beginner-friendly guide to high-performance scale-out networking HPC and AI Environments Quick Reference, Glossary, FAQ Quick reference cheat sheet Table 25. InfiniBand/RDMA cheat sheet Term Meaning IB InfiniBand RDMA Remote Direct Memory Access HCA Host Channel Adapter QP Queue Pair SQ Send Queue RQ Receive Queue CQ Completion Queue WR Work Request WQE Work Queue…
-
A beginner-friendly guide to high-performance scale-out networking HPC and AI Environments Use Cases, a Small Cluster Walkthrough and Scaling Up Common HPC workloads Table 20. HPC workload examples Workload Why the fabric matters Computational fluid dynamics Domain decomposition exchanges boundary state every iteration Weather/climate modelling Large distributed grids require frequent neighbor/global communication Molecular dynamics Many…
-
A beginner-friendly guide to high-performance scale-out networking HPC and AI Environments InfiniBand versus Ethernet/RoCE – and versus NVLink InfiniBand versus conventional TCP/IP Ethernet Conventional Ethernet is an extraordinarily flexible general-purpose network. InfiniBand is a specialized fabric with native RDMA transports, subnet management, credit-based flow control and an HPC-oriented software ecosystem. The comparison is therefore not…
-
A beginner-friendly guide to high-performance scale-out networking HPC and AI Environments Resiliency, Failure Behavior and Security Network redundancy is not application fault tolerance A fabric can provide multiple physical paths and still lose an application when a node or rank fails. Redundant links help the network survive certain link/switch failures; distributed applications must separately decide…
-
A beginner-friendly guide to high-performance scale-out networking HPC and AI Environments The HPC Software Stack: MPI, UCX, Verbs and OFED From application to fabric Figure 17. Typical HPC communication software stack MPI (Message Passing Interface) is the application-facing programming model used by many HPC codes. Implementations such as Open MPI or MPICH can use Unified…
-
A beginner-friendly guide to high-performance scale-out networking HPC and AI Environments Addressing, Subnets and the Subnet Manager GUID, GID and LID Figure 9. Identity hierarchy Table 9. GUID versus GID versus LID Identifier Scope/purpose Beginner mental model GUID Globally unique hardware/port identity Who is this object? GID 128-bit global identifier associated with a port/context How…
-
A beginner-friendly guide to high-performance scale-out networking HPC and AI Environments Inside an InfiniBand Fabric: Architecture, Components and Speeds High-level architecture Figure 3. Basic two-tier InfiniBand fabric Compute nodes attach through Host Channel Adapters (HCAs). Switches forward packets across paths programmed by the subnet-management function. The Subnet Manager (SM) discovers the topology, assigns local identifiers,…
-
A beginner-friendly guide to high-performance scale-out networking HPC and AI Environments The one-sentence summary InfiniBand is a specialized switched fabric for moving data between distributed compute resources with very low latency and high throughput; RDMA lets applications arrange much of that movement directly between registered memory regions with far less CPU and kernel involvement on…