Training a large language model does not stall for lack of compute. In a training cluster thousands of graphics cards spend a good part of their time waiting for each other, and the culprit is almost never inside the chip. It is in the path that joins them: after every layer of the neural network each card holds a slice of the result and all of them need the whole. That exchange moves in gigabytes per second, and it decides whether the cluster runs at 40 percent or at 90 percent of its theoretical capacity.
A sum where everyone depends on everyone
When a model does not fit in the memory of a single GPU it is split in two ways. In tensor parallelism one layer is cut into pieces and each card computes a fragment of the weight matrix; in data parallelism each card trains on a different batch of the same corpus and at the end the gradients have to be averaged. Both paths end in the same collective operation: the all-reduce, a distributed sum in which every node contributes its share and every node receives the complete result. The most efficient way to run it on real hardware is a ring: the GPUs are ordered in a circle, each passes its chunk to the next and the next accumulates it. With N nodes each one moves 2(N-1)/N times the size of the vector, that is, almost twice the original data instead of N times. That detail, which looks like a textbook exercise, is what makes training on thousands of cards viable.
Why an ordinary TCP connection is not good enough
The classic network stack was designed for reliability, not for latency. Every message crosses the kernel, is copied into system buffers, triggers interrupts and wakes processes. The typical cost is between ten and thirty microseconds per operation, with a variance that in a large cluster turns into chained waiting: if each layer chains dozens of messages, the delay eats milliseconds that no GPU can recover. On top of that, moving data between card memory and host memory keeps CPU cores busy that should be preparing the next batch. On a node with eight GPUs that accounting can leave the training job itself without CPU time.
RDMA: writing into another machine memory without telling its processor
Remote Direct Memory Access changes the model: instead of sending a message to a process, a network card writes directly into a memory region of the machine on the other side. To make it safe, every buffer is registered beforehand (memory registration) and pinned in RAM so the system cannot move it, with a local and a remote key that authorise the access. Traffic is organised in queue pairs, a send queue and a receive queue, and completions are polled from completion queues. There are one-sided operations —RDMA WRITE, RDMA READ and atomics such as fetch-and-add— that the receiver does not have to serve: its CPU never finds out. Latency drops to one or two microseconds and, because there are no interrupts, programs do not wait: they poll a flag in memory in a loop and keep computing. It is the same working style the kernel itself uses to sleep and wake threads, taken to the network.
Two motorways: InfiniBand and Ethernet with RoCEv2
There are two ways to lay that fabric. InfiniBand is a network of its own, with its own switches, offering 400 Gb/s per port today (NDR) and 800 Gb/s (XDR), with adaptive routing and hop latencies below one microsecond; it is the choice of the largest AI clusters. The alternative is RoCEv2: RDMA encapsulated over Ethernet and UDP, using standard network gear, cheaper and easier to operate. In exchange it demands a lossless network, and that is achieved with two delicate mechanisms: PFC (priority flow control), which pauses a link before its queue overflows, and ECN/DCQCN, which marks packets so senders slow down. Badly configured, they become the worst enemy: one PFC pause can propagate across the whole network and freeze the entire cluster.
GPUDirect: the cable that reaches into card memory
Without help, data travels through GPU, host RAM, network card and back, with two PCIe copies on each leg. GPUDirect RDMA lets the network card write straight into the GPU HBM memory, skipping the server RAM. Inside the node the shortcut is NVLink and NVSwitch, which connect the cards to each other at speeds close to a terabyte per second per GPU, far above what the PCIe bus can do. Between nodes, the usual topology is rail-optimized: every GPU has its own network card and all the cards in the same position hang from the same switch, so traffic is spread across rails instead of fighting over a shared link.
Whoever splits the work decides the performance
Collective libraries —NCCL in the NVIDIA world— pick the algorithm in real time. With small messages latency rules and a tree wins; with large vectors bandwidth rules and a ring wins; when there are many GPUs in the same chassis, the reduction is done first inside over NVLink and then across nodes. The choice matters because performance is set by the slowest link: a single degraded connection forces the whole group to wait on every iteration. That is why frameworks overlap communication with computation, launching the reduction of one layer while the next is still being computed, and why network testing and checkpointing have become everyday routine.
The failure nobody sees: a degraded fabric
In these clusters the dangerous thing is not a broken cable, which is spotted at once, but the cable that works badly. A link with bit errors silently corrected by FEC, a PFC pause counter creeping up or a switch that drops one packet in a thousand degrade training without raising any visible alarm: the job keeps running, only 15 percent slower for weeks. Monitoring is done with per-lane counters, error rates and latency maps between pairs of nodes, and in practice teams reserve maintenance windows to validate the fabric with bandwidth and collective tests before launching a long training run.
Where it is heading: 800G, reduction inside the network and Ultra Ethernet
The next steps are three. First, more speed: 800 Gb/s and 1.6 Tb/s links, and fibre optics displacing copper inside the rack itself. Second, offloading work onto the network: some designs let switches partially add up the data while they carry it, so only the final result reaches the GPU and the traffic is halved. Third, a new transport: the Ultra Ethernet consortium is working on a protocol tailored to these workloads, with modern congestion control, aiming for the reliability of InfiniBand without giving up the Ethernet ecosystem. In the end, the metric that decides which cluster is better is no longer how many trillion operations per second a GPU does, but how much bandwidth it has towards the others and how many times per second it needs to use it.
