Every time you type something into a search engine, the answer arrives in a fraction of a second. That speed is no magic trick: it relies on work that happened long before you pressed Enter. By the time you submit your query, the web has already been read, split into pieces and sorted into a structure called an inverted index.
Searching directly over the billions of pages that exist would be impossible: you would have to open each document, read it entirely and compare it with your question. That would take minutes, not milliseconds. The key is to flip the perspective: instead of storing each page with its words, you store each word with the pages where it appears.
The crawlers that read the web before you do
The process starts with crawlers (or spiders), programs that walk the internet following links from page to page. A crawler keeps a queue of pending URLs: it downloads a page, extracts its links, adds them to the queue and repeats. Starting from a few initial seeds, it can discover practically the whole public web.
Crawlers respect the robots.txt file, a document each site publishes to say which sections may be visited and which may not. They also follow a politeness policy: instead of hammering a server with hundreds of requests per second, they space their visits so as not to bring it down. And they revisit pages periodically, because the web changes constantly.
The inverted index: the star of the system
What the crawler downloads is not stored as-is. Each document is tokenized: the text is split into words or tokens, lowercased and passed through normalization techniques such as stemming (reducing “running”, “run” and “ran” to the root “run”). The goal is that different forms of the same idea point to the same term.
From those tokens the inverted index is built. It is a dictionary where every term in the vocabulary points to a posting list: the set of documents containing that word, together with useful data such as how often it appears and its position within the text. Frequency and position turn out to be gold when ranking results.
These lists are not stored carelessly. Document identifiers are sorted and compressed with techniques such as delta encoding (storing the difference between consecutive identifiers instead of the whole number), which lets a giant index fit in RAM. And skip pointers are added — jump pointers that speed up intersections between lists without walking them entirely.
From the list to the ranking: TF-IDF and BM25
Finding the documents that match the query is only half the job. The other half is ordering them: which one is most relevant? This is where scoring models come in. The classic one is TF-IDF, which combines two measures. Term frequency (TF) counts how often the word appears in the document: the more appearances, the more relevant. But there is a catch: if the word appears in nearly every document, it stops being useful for telling them apart. Inverse document frequency (IDF) penalizes overly common terms such as “the” or “and”.
The modern standard is BM25, an evolution of TF-IDF that adds two important corrections. On one hand it normalizes by document length: a short page containing the term deserves more credit than a long one that mentions it in passing. On the other it introduces frequency saturation: a term appearing five times contributes quite a bit more than once, but fifty appearances do not multiply relevance by ten.
Ranking is not left to keyword matching alone. Search engines add authority signals: how many other pages link to a document (the original idea behind PageRank), the text of those links, click data from previous users and the freshness of the page. The final result is a combined score that decides the order of the results.
How your query is processed in milliseconds
When you run a search, the engine tokenizes your query just as it did with documents. Then it locates the posting list of each term in the index. If you search “algorithm guide”, it intersects the “algorithm” list with the “guide” list (using the skip pointers to move fast) and keeps the documents present in both.
The index is usually sharded: split by ranges of documents or by terms and distributed across hundreds of servers running in parallel. Each shard returns its best results and a central coordinator merges them into a single list. That is why a query touching millions of documents can be answered in a handful of milliseconds.
Add caching to that: the most popular queries and their results are stored to be served instantly, and responses are replicated on servers close to the user to cut latency. Freshness is handled with a “hot” index updated in real time for news or social pages, combined with the full index rebuilt in the background.
The illusion of instant search
Next time a search returns results in the blink of an eye, remember: nobody went searching “live”. Behind it is an infrastructure that already read, split and sorted a huge fraction of the internet before you even asked your question. The inverted index and the ranking are the two engines that turn billions of pages into an almost instant answer. The magic, as always, is good engineering.
