On 4 October 2021, Facebook, Instagram and WhatsApp vanished from the internet for nearly six hours. There was no attack, no power failure, no fire in a data centre. Their servers stayed up and their DNS resolvers kept answering queries from outside. What broke was something far quieter: for a few minutes, the company’s routers withdrew the BGP announcements that told the rest of the world “this network is here”. The internet then did the only thing it knows how to do with a network that is no longer announced: it stopped sending it packets.
That episode sums up BGP (Border Gateway Protocol) rather well, the protocol that has been running since 1989 and remains the backbone of routing between operators. It does not move data; it moves promises about where networks live. And every packet’s arrival depends on those promises.
A map nobody draws
The internet is not a cloud: it is a federation of roughly 76,000 autonomous systems, or ASes. An AS is a set of IP networks run by a single entity — a carrier, a university, a company, a cloud provider — with one common routing policy, identified by a number: the ASN. Telefonica is AS3352, Google is AS15169, Cloudflare is AS13335.
Each AS announces to its neighbours the address blocks it controls, the so-called prefixes (say 203.0.113.0/24). Those neighbours pass them on to theirs, and so on until the prefix is visible across virtually the entire global table, which today holds around a million IPv4 entries and more than 200,000 IPv6 ones. Nobody has the full map, by design: each router only knows the route its neighbours told it about and the best exit for each destination according to its own rules.
One TCP session and four messages
Technically, BGP is a path-vector protocol. Unlike OSPF or IS-IS, which compute the lowest-cost path inside a domain, BGP optimises nothing: it exchanges routes and applies policy. Two routers establish a session over TCP port 179, swap OPEN messages to negotiate version, ASN and capabilities, and then keep it alive with a KEEPALIVE every 60 seconds and a 180-second hold timer. If the timer expires, the session drops and every route learned through it is withdrawn at once.
There are only four message types: OPEN, UPDATE, KEEPALIVE and NOTIFICATION. All the real work happens in UPDATE, which carries two parts: the NLRI (Network Layer Reachability Information) — the prefixes being announced or withdrawn — and a set of attributes describing the route. That is where the decisions actually live.
The attributes that rule
When a router receives several routes to the same prefix, it does not pick the fastest one: it walks an ordered list of criteria. First comes local weight, which exists only on the router itself. Then LOCAL_PREF, the preference inside the AS: the higher, the better. Then AS_PATH length, the list of autonomous systems the announcement has traversed; the shortest wins, which does not mean the fastest, only the one crossing fewer operators. Further down sit route origin, MED (Multi-Exit Discriminator, a hint for choosing between two entry points into the same neighbour), eBGP over iBGP preference, and the internal metric to the next hop.
That order explains why the internet does not follow the geometrically short path. A carrier may prefer sending traffic to another continent because it has a cheaper peering agreement, or because it wants it to exit through a particular exchange point. Economics is written into the attributes. Communities — 32-bit tags operators pass to each other — add another layer: they mark routes as “do not announce to customers” or “prefer in Europe”, and they underpin almost all real-world traffic engineering.
Anycast: the same address in two hundred places
One of BGP’s most elegant applications is anycast: announcing the same prefix from many different locations. Because each router picks a single route per prefix, a user’s traffic ends up at whichever announcement point their carrier considers best, which in practice is usually the closest in hop count. That is how 8.8.8.8 and 1.1.1.1 work, along with most DNS root servers and content delivery networks: one IP address, hundreds of sites, and a DDoS attack spread across all of them instead of concentrated on one.
The flip side is that a badly filtered anycast announcement can pull traffic from half the planet towards a single point. The number matters: a /24 announced by mistake from one edge router can hijack all traffic for that network within seconds.
When someone announces what is not theirs
BGP does not verify ownership of a prefix. If an AS announces a block it does not own, its neighbours believe it and propagate it. In 2008, Pakistan Telecom tried to block YouTube nationally by announcing its range; the announcement leaked to an international carrier and YouTube became unreachable for roughly two hours across much of the world. In 2019, a Verizon misconfiguration propagated thousands of foreign prefixes for a few minutes, including Cloudflare and Amazon blocks, causing notable outages for third parties. In 2021, Facebook’s incident was the voluntary version of the same mechanism: an internal command withdrew the announcements and the company disappeared from the routing table.
One detail makes it worse: the longest-prefix-match rule. If you announce 203.0.113.0/24 and someone else announces 203.0.113.0/25, the second wins for half the block, even with a longer AS_PATH. Splitting a prefix into smaller pieces is enough to divert traffic without touching a single attribute.
RPKI: signing the origin
The main defence deployed in recent years is RPKI (Resource Public Key Infrastructure). Regional internet registries — RIPE NCC, ARIN, LACNIC — issue X.509 certificates binding address blocks to ASNs, and the holder publishes a ROA (Route Origin Authorization): a signature stating “only AS15169 may announce this prefix, up to this maximum length”. Routers performing Route Origin Validation (ROV) classify each route as valid, invalid or not found, and operators can drop the invalid ones.
It works, with two big caveats. The first is coverage: even today only about half the announced space has a ROA, and many operators validate but do not drop, for fear of breaking legitimate traffic. The second is conceptual: RPKI protects the origin, not the path. An attacker controlling a legitimate AS can still manipulate AS_PATH, and BGPsec — the extension meant to sign the full route — remains essentially undeployed because of its computational and coordination cost.
Why it is not getting fixed
BGP is nearly forty years old and carries a historical compromise: it was designed so that networks run by organisations that do not trust each other could exchange routes with minimal coordination. Changing it requires tens of thousands of operators to update software and policy at the same time. Until then, security advances in layers: prefix filters and max-prefix limits on sessions, IRR databases to check what each AS should be announcing, RPKI for origin, and shared practices such as those promoted by MANRS.
The uncomfortable conclusion is that the internet we use does not rest on a system that guarantees truth, but on cooperation and careful configuration by tens of thousands of strangers. It works surprisingly well. But every time someone mistypes a prefix list, the whole map gets redrawn for a few minutes.
