When someone says “the call looks terrible”, they usually blame the carrier, the router or the laptop. The interesting part is what happened a few hundred milliseconds earlier: the software holding that video call together has taken dozens of decisions about which information deserves to reach you and which information is thrown away without anyone noticing. A video call is not a continuous stream of data: it is a permanent negotiation with loss.
What follows is an ordinary WebRTC call, the same one used by Meet, Teams, Zoom in its web mode, Discord or Jitsi. The stack is public, standardised by the IETF and the W3C, and it has spent years solving a problem that looks simple and is not: carrying real-time audio and video over a network that guarantees absolutely nothing.
Before you can talk, you have to find each other: ICE, STUN and TURN
Two browsers do not connect just because they are on the internet. Each one sits behind a NAT (the mechanism that translates the private addresses of a home or an office into a single public address), so the first job is to discover which path can work. That is what ICE (Interactive Connectivity Establishment) is for, and it does not connect anything by itself: it gathers candidates, that is, possible addresses and ports, and then tests which ones actually work.
There are three kinds of candidates. Host candidates are the local addresses of the machine. Server-reflexive candidates come from asking an external STUN server: the server replies with the public address and port as seen from the outside, which is what the other side needs to know. And relay candidates come from a TURN server, which acts as an intermediary and forwards traffic when both endpoints sit behind symmetric NATs and no direct path exists. In real deployments, the share of calls that need that intermediary is usually estimated at between 10% and 20%, and those are precisely the expensive ones: the server pays for the bandwidth in both directions and adds latency.
The formal negotiation uses SDP (Session Description Protocol): a text in which each side lists codecs, ciphers, addresses and parameters as a=, m= and c= lines. One offer and one answer are enough to agree on what will be sent. Curiously, that dialogue is not part of the standard: it travels however you like, usually over a WebSocket, and it is called signalling. WebRTC encrypts the media but leaves signalling to the application.
Twenty milliseconds per envelope
Audio travels over RTP (Real-time Transport Protocol), and the unit is not the second: it is blocks of 20 milliseconds. Fifty envelopes per second in each direction, each with a 12-byte header, a 16-bit sequence number and a timestamp in 48 kHz units. The sequence number is the key piece: when a packet arrives too late, the receiver does not wait for it indefinitely, it declares it lost and moves on. That small act of giving up is what makes conversation possible.
Alongside it runs RTCP, the control channel: receiver reports stating how many packets were lost, how much delay variation (jitter) is being measured and what the round-trip latency is. Without those reports the sender would be flying blind, and all the adaptation described below leans on them.
Opus, a codec that changes shape
Audio is compressed with Opus, a hybrid codec combining two engines: SILK, aimed at speech (linear prediction, designed for vocal signals), and CELT, aimed at music and wideband signals (a transform in the frequency domain). It can run between 6 and 510 kbps, with frames from 2.5 to 60 ms, and it adjusts its mode and its rate on the fly without breaking the stream.
Two details matter more than they seem. The first is in-band FEC: alongside the current frame, the codec can send a low-rate compressed copy of the previous one, so that if a packet is lost the receiver can rebuild something intelligible instead of a hole. The second is DTX together with voice activity detection: during silences almost nothing is sent, only comfort frames. In a normal conversation, one of the two speakers is silent more than half the time, and that shows up in the data bill.
The cushion that buys time: the jitter buffer
Packets do not arrive at exact 20 ms intervals. They arrive in bursts, sometimes out of order, with delays that keep changing. To turn that disorder into continuous audio there is the jitter buffer, a small queue with an adaptive target delay, typically in the 30 to 100 ms range. If the network turns jumpy, the queue grows to absorb the jitter; if it settles, the queue shrinks so as not to give away latency for free.
When the queue runs short, the engine performs a juggling act: it speeds up fragments with time stretching (compressing time without changing pitch), merges similar frames or, if there is nothing left to use, conceals by extrapolation, that is, it invents a plausible continuation from the last good fragment. That is what you hear as a small metallic stutter when the wifi wobbles. It is not a codec failure: it is a calculated decision to avoid a silence that would sound worse.
Video is not sent: it is sent in layers
Here is the big trick. Instead of producing a single video stream that would have to be re-encoded on demand, the sender generates several qualities at once. With simulcast it encodes two or three independent versions simultaneously (say 180p, 360p and 720p) and sends them in parallel; with SVC (Scalable Video Coding), available in VP9 and AV1, it encodes a layered structure inside a single stream: spatial layers, which add resolution, and temporal layers, which add smoothness.
In the middle there is usually an SFU (Selective Forwarding Unit), a server that re-encodes nothing: it merely chooses, for each participant, which layers to forward. With twenty people in a room, re-encoding twenty streams would be unfeasible on CPU; forwarding selectively is cheap. The other model is the mesh, where everyone sends to everyone else: it works with three or four people and collapses as the number grows, because upstream bandwidth multiplies.
Layers are also the currency when the network degrades. Dropping a layer does not require re-encoding: it means stopping the sending of data that was already encoded. And there is a fundamental asymmetry: a key frame (an intra frame, one that does not depend on previous ones) weighs several times more than a differential one, so requesting a new one — that signal is called PLI or FIR — is expensive and is done sparingly, not every time a packet is lost.
Measure before deciding: bandwidth estimation
All of the above needs to know how much fits along the path. That is the job of congestion control, and in the WebRTC world the reference algorithm is GCC (Google Congestion Control). Its idea: if packet arrival times start to grow steadily, a queue is filling up in some router, and that is a sign that we are pushing past the real capacity. A statistical filter over arrival timestamps — which the receiver returns to the sender in the transport-cc reports — estimates that trend; the controller combines it with observed loss and produces a bitrate target.
That target is not applied at once: a pacer spreads packets over time so as not to cause bursts that fill queues, and controlled tests (probing) deliberately raise the rate to see whether the network can take more. The old REMB message, with which the receiver dictated the bitrate to the sender, fell out of use in favour of this per-packet feedback system. The outcome of all of it is a figure that changes every few hundred milliseconds and that decides, in practice, which video layers are sent and which are not.
The order of sacrifice
When bandwidth narrows, not everything degrades at once: an order is applied. First the high spatial layers are abandoned, then resolution, then frame rate and, last of all, the quality and redundancy of the audio. The reason is simple: in a conversation, understanding the words matters more than seeing the face clearly. A 32 kbps Opus stream with FEC remains perfectly intelligible on a connection that no longer supports decent video.
That is why, when the network goes through a bad patch, what you see is not a blurry image struggling onwards: it is the last good picture, frozen, while the sound keeps going. It is literally an old frame held on screen while the receiver waits for a new one it can decode. Retries do exist (receivers can ask for lost packets via NACK), but they operate on a time budget: if the retransmitted packet is going to arrive after its moment has passed, it is not requested. Retransmitting late is as useless as not retransmitting. And when it is requested, redundancy is sent: some schemes duplicate the previous packet at a bandwidth cost of between 20% and 50%, depending on how much protection is applied.
And if the network dies completely
Worse than loss is the path ceasing to exist: someone switches networks, enters a lift, a route goes down. ICE can restart, negotiating new candidates without tearing down the call, and it is common for the connection to reattach through the TURN relay over TCP or TLS on port 443, the one almost no network blocks because it is the same port any website uses. The key is not to wait for the user to complain: endpoints periodically verify that the path is still alive (consent freshness checks), so a dead path is abandoned quickly instead of feeding a black hole.
Encryption always, end-to-end optional
In WebRTC encryption is not optional: keys are negotiated with a DTLS handshake and the media travels in SRTP with AES-GCM. That is, the SFU forwarding the media usually sees the content, because it decrypts and re-encrypts. End-to-end encryption exists (SFrame, and in the browser the encoded transform API) and is in production in some services, but it has a clear technical price: if the server sees nothing, it cannot transcribe, cannot record in the clear and cannot choose layers with the same freedom. Security and functionality pay for each other.
What is routine in 2026
Transport has kept modernising: AV1 with SVC in browsers, QUIC and WebTransport based ingest, simplified publishing protocols for desktop encoders and, in audio, neural codecs that achieve intelligible speech at rates that seemed like a toy years ago. Echo cancellation and noise suppression are already done with trained models that tell speech apart from a keyboard, a fan or a barking dog. None of that changes the underlying point: call quality still depends on a bandwidth estimate and on the decision to throw away what does not fit.
The engineering of throwing things away
A video call that survives a mediocre network is not a feat of brute speed but of discipline. Every 20 milliseconds there is an audio packet that arrives or does not, a queue being adjusted, a video layer being abandoned and an estimate being corrected. The feeling that “it looks fine” is, in fact, the sum of hundreds of well-made concessions. The next time the picture freezes while you keep hearing the other person, you will know nothing has broken: it is protecting what really matters.
