When you type a question into a chat and the model answers in seconds, behind it is one of the most complex software architectures that exist. Large language models (LLMs) do not “search” for the answer in a database: they generate it token by token, word by word, calculating probabilities over tens of billions of parameters. This article explains how they manage to do it.
From text to numbers: the tokens
A language model does not understand letters, but tokens: text fragments of between one and several characters. The word “internet”, for example, can be split into two tokens. A tokenizer (the component in charge of this conversion) transforms each fragment into an integer that points to a vocabulary table. The size of that vocabulary usually hovers around 100,000 elements. With this, the whole sentence ends up converted into a sequence of IDs that the model can process mathematically.
The heart: the transformer
The architecture that makes all this possible is called transformer, first described in 2017 in the paper “Attention Is All You Need” by Google. Its key piece is the self-attention mechanism: each token in the text looks at all the other tokens in the sequence and computes how much “weight” to give to each one. Thus, when the model processes the sentence “the cat crossed the street because it was in a hurry”, it understands that “hurry” refers to the cat, not the street.
These weights are calculated by multiplying three vectors per token: query (what I am looking for), key (what I offer) and value (what information I contribute). The product between query and key decides the attention; then a normalization (softmax) is applied that converts those values into probabilities. This whole operation is repeated in multiple attention “heads” (multi-head) and in dozens or hundreds of stacked layers.
Parameters, memory and compute
The parameters are the internal weights adjusted during training; a modern model can have between 70 billion and more than a trillion. Each token processed requires a forward pass through all the layers, multiplying and adding matrices of thousands of dimensions. That is why GPUs (graphics processing units) or specialized accelerators like Google’s TPUs are used: their massive parallelism is ideal for these matrix operations.
Memory is a bottleneck: keeping the intermediate states of each layer requires a lot of RAM. That is why quantization techniques emerged, which reduce the numerical precision of the parameters (for example, from 16 bits to 8 or 4 bits per value) so the model fits in less memory at the cost of a minimal loss of quality. There are also context windows: the maximum number of tokens the model can “see” at once, which in current models ranges from 8,000 to more than a million.
Generating text: sampling and temperature
After the last layer, the model returns a logit (a score value) for each token in the vocabulary. A softmax step converts those logits into a probability distribution over 100,000 possible next tokens. But the model does not always choose the most probable one: it uses weighted random sampling. The temperature parameter controls how “creative” the choice is: with low temperature it becomes deterministic and predictable; with high temperature, more varied and risky.
Each chosen token is added to the input sequence and the model repeats the process for the next one, until it generates a special end token. That is why a long answer involves thousands of these passes, which explains the latency (the waiting time) you notice on screen.
How they learn: pretraining and fine-tuning
The model’s knowledge is not written by hand: it comes from pretraining on huge amounts of text (billions of words from books, web pages and code). The goal is simple and powerful: predict the next token of each fragment. When it fails, an algorithm called backpropagation adjusts the parameters with an optimization technique (for example, Adam) to reduce the error. After billions of these corrections, the network ends up capturing statistical patterns of language, syntax and even reasoning.
Then comes fine-tuning: the model is trained again, now with the learned weights, on smaller and higher-quality data, often with human feedback (RLHF, reinforcement learning from human feedback). This aligns the behavior so it responds in a useful, safe and coherent way.
The future: more context, less cost
The current race focuses on expanding context windows, reducing inference cost and making smaller but capable models (SLMs, small language models) that run on the phone itself. Mixture-of-experts models (MoE), which activate only part of the network per token, promise the intelligence of a giant model at the cost of a small one. The result: the same technology behind your favorite chat, increasingly faster and more accessible.
