Look at a video game scene: millions of triangles, light reflecting, smoke drifting. It looks like the graphics card is “drawing”. It is not: it is computing. Every pixel on your screen is the result of an arithmetic operation — in fact, of thousands. And the secret is not magic, but an architecture that sacrifices the individual cleverness of each core to multiply their number beyond anything a CPU will ever reach.
Two philosophies of computation: latency vs. throughput
A modern CPU has between 8 and 24 cores, each able to run very different instructions, with branch prediction, out-of-order execution and huge caches. It is built for latency: solving one complicated task as fast as possible. A GPU, on the other hand, chases throughput: solving many simple tasks at the same time. A GPU such as the RTX 4090 packs more than 16,000 cores; the trick is that they all execute the same operation at once, each on different data.
SIMT: many threads, one instruction
The GPU execution model is called SIMT (Single Instruction, Multiple Threads). Cores are grouped into units of 32 threads called warps: all 32 execute the same instruction simultaneously, but each one operates on its own data. It is an evolution of the old SIMD (single instruction, multiple data), with the difference that each thread has its own registers and its own state, so it can branch independently. If the 32 threads take different paths, the warp diverges and executes each branch separately: that is why shader code that avoids if statements splitting many threads usually runs faster.
The image pipeline: from triangle to pixel
The classic flow starts with the vertex shader, which transforms every 3D vertex with projection matrices; then comes the rasterizer, a fixed-function unit (a dedicated circuit, not programmable) that decides which pixels each triangle covers; and it ends in the fragment shader, which computes the final color of each pixel by combining lighting, textures and shadows. Shaders have been unified for years: the same programmable cores run vertex, fragment, geometry or compute shaders depending on the workload. Modern GPUs also add specialized fixed units: RT cores, which accelerate ray tracing by traversing BVH structures (bounding volume hierarchies) to find out which ray hits which object, without occupying the programmable cores.
The bottleneck is not arithmetic: it is memory
A GPU core computes a multiplication in one cycle; the problem is feeding it. That is why VRAM (video memory) is so wide: the RTX 4090 moves 1,008 GB/s with GDDR6X, and server GPUs use HBM (High Bandwidth Memory), which stacks chips in layers and exceeds 3 TB/s. To hide memory latency, the GPU relies on occupancy: instead of waiting for a warp to fetch its data, it switches to another warp that is ready. It is latency hiding through massive context switching, and it requires the compiler to allocate enough registers and shared memory per block.
Tensor cores: why AI lives on GPUs
Training a model like GPT means multiplying huge matrices, over and over. GPUs added dedicated circuits for that: tensor cores, which run matrix multiply-accumulate in reduced precision (FP16, BF16, TF32 and even FP8). A Hopper-generation tensor core does thousands of operations per cycle in a single matrix instruction, while 64-bit arithmetic, needed for science, is cut down to a far slower rate. The result: a modern GPU can exceed one petaFLOP (a thousand trillion operations per second) in FP8 precision.
How data reaches the chip
The CPU talks to the GPU through the PCIe bus: an x16 PCIe 5.0 slot transfers about 63 GB/s in each direction, an order of magnitude below VRAM. That is why datacenter GPUs connect to each other with NVLink (up to 900 GB/s per GPU in the newest systems) and to the CPU with proprietary bridges. It is also why moving data between system memory and VRAM is the most expensive operation in the whole compute pipeline: AI frameworks such as CUDA and ROCm spend a good part of their time trying to keep tensors already on the GPU.
The future: more fixed units and fewer watts per operation
The trend is clear: every generation adds more specialized circuitry — tensor cores, RT cores, video decoders, path-tracing units — and leaves the programmable cores for whatever does not fit a fixed pattern. Energy efficiency (operations per watt) has become the guiding metric of AI data centers. Next time you watch a photorealistic scene in motion, remember: there is no brush, only additions and multiplications performed at the speed of light by an army of tiny calculators.
