Every time you type into a chatbot, you have switched on a machine that does not “remember” your conversation the way a person does. It does not re-read it: it recomputes it. And that recomputation has a price that grows in a very specific way with every word you add. Understanding that price is understanding why language models slow down, why API bills climb steeply with long contexts, and why half the industry is researching ways to make attention cheaper.
A model does not store sentences, it stores vectors
When text enters a transformer — the architecture behind today’s language models since 2017 — the first thing that happens is tokenisation: the text is split into pieces called tokens, which are not exactly words. “House” may be one token; “casually” is probably three. Each token becomes a vector of, say, 4096 numbers, an embedding, which is a coordinate in a space where words with similar meanings land close to one another.
But a fixed vector is not enough: the word “bank” means different things in “I sat on the river bank” and “I went to the bank for a loan”. Meaning depends on context, and that is where the attention mechanism comes in.
Attention: every token asks all the others
In each layer of the network, every token produces three vectors called Q, K and V (query, key and value). The classic analogy is a fuzzy database lookup: a word’s Q vector is the question that word is asking (“who is shaping my meaning?”), the K vectors of the other tokens are the labels that answer that question, and the V vectors are the content that gets blended according to how well they match.
The score is computed as a dot product between Q and each K, divided by the square root of the vector dimension so the numbers do not blow up, then passed through a softmax function that turns it into weights summing to one:
attention = softmax(Q · Kᵀ / √d) · V
The important detail is the shape of that expression. To predict each token, the model compares that token with all the ones before it. With 10,000 tokens in the context, each of them faces the other 10,000. Compute cost grows with the square of the token count: this is the quadratic O(n²) cost of attention. Doubling the text on screen multiplies the comparisons by four.
The KV cache, or why context eats memory
Recomputing everything from scratch for every new word would be absurd, so inference engines use the KV cache: the K and V vectors of previous tokens are already computed, stored and reused. During generation, the model only computes the vectors of the newly arrived token and appends them to the cache.
That cuts the work but creates another problem: the cache grows linearly with context length and has to stay in GPU memory for the whole generation. Its size in bytes is roughly:
2 · n_tokens · n_layers · n_heads · d_head · 2 bytes
With an 80-layer model, 64 attention heads and 128 dimensions per head, each token takes about 2.6 megabytes. A hundred thousand tokens of context will not fit on an 80 GB GPU: that is why attention has become a memory problem, not just a compute problem. There are also two very different phases: prefill, where the model processes all your text at once and the GPU has plenty of arithmetic to do but is limited by the memory it moves; and decode, where words are generated one at a time and the GPU sits nearly idle waiting to read the cache.
The four tricks that are working
Grouped-Query Attention (GQA). If several attention heads share the same K and V vectors instead of each owning a set, the cache shrinks to a fraction of its size. This is why today’s models can attend to contexts of hundreds of thousands of tokens without dying of memory exhaustion.
FlashAttention. The algorithm never writes the full n×n score matrix into the GPU’s fast main memory (HBM), which is the expensive part: it splits it into small blocks, processes them in the chip’s on-die memory (SRAM) and accumulates the result. The maths does not change, the memory traffic does, and that alone multiplies speed several times over.
Sliding-window attention. Many models only let each token look at the last few thousand tokens rather than all of them. The cost becomes roughly linear, at the price of letting very old information fade.
Prefix reuse. If a prompt always starts the same way (a manual, a contract, the system instructions), that prefix’s cache is stored and reused across requests. This is the prompt caching the APIs offer: you pay much less for the repeated beginning of each query.
What this means when you pay the bill
Providers charge per input token and per output token, and the input price often multiplies once the context passes a certain threshold. Behind that number sits the quadratic cost we described and the memory the cache occupies. Pasting a 200-page PDF into a conversation does not just cost more: it makes every later reply slower and more expensive, because everything generated afterwards drags that context along.
The next time a chatbot seems to lose the thread of a long conversation, the reason is probably not that it “gets distracted”. It is that attending to 150,000 tokens forces the attention weight to be spread across far more candidates than were used during training, and the model is doing what it can with the memory that fits on the graphics card.
