It is half past three in the afternoon in Madrid and your laptop has just asked a server in Virginia what time it is. Twenty minutes have passed between the reply and the next one; by the time it finishes talking to that server, the system clock will have moved by a couple of milliseconds and you will notice nothing. Behind that apparently trivial adjustment lies one of the oldest and most delicate pieces of infrastructure on the internet: a hierarchy of clocks that corrects itself, that fights the drift of quartz crystals, and that in its industrial versions reaches resolutions measured in nanoseconds.
The problem: a quartz crystal is never right
Every computer carries a quartz oscillator producing pulses at a nominal frequency, typically 32,768 Hz for the real-time clock (RTC) and tens or hundreds of megahertz for the clock that counts processor cycles. That crystal has a manufacturing error and, worse, it depends on temperature: a cheap oscillator drifts between 10 and 100 parts per million. In real time, 50 ppm mean 4.3 seconds of deviation per day and more than twenty minutes per year.
On top of that, the operating system does not distinguish a single notion of time but several. CLOCK_REALTIME is the wall clock, the one that can jump forward or backward when someone adjusts it; CLOCK_MONOTONIC only moves forward, which is why it is used to measure durations; CLOCK_BOOTTIME keeps counting through suspension; and CLOCK_TAI is the atomic scale without leap seconds. Confusing them is the cause of a good share of subtle bugs in distributed systems: comparing two timestamps taken from different clocks means nothing.
How to measure a server’s clock without trusting it
NTP (Network Time Protocol), defined in RFC 5905, solves the problem with four timestamps. The client sends a request at T1; the server notes arrival at T2 and departure at T3; the client receives at T4. Two fundamental quantities come out of that:
- Offset = ((T2 − T1) + (T3 − T4)) / 2. It estimates the skew between the two clocks.
- Delay = (T4 − T1) − (T3 − T2). That is the round-trip time minus how long the server took to answer.
The trick is that if the network were perfectly symmetric, the offset would be exact without any need to synchronise the outbound clock with the inbound one. Since the network is not symmetric, NTP adds two statistical defences: the clock filter algorithm, which out of every eight samples keeps the one with the lowest delay, and the intersection algorithm, inherited from Marzullo’s algorithm, which discards servers that lie or drift badly. A client can query four or five servers and settle on the confidence interval common to all of them.
The accuracy achievable on the open internet is a few milliseconds; on a local area network with well-connected servers it drops to tens of microseconds. And every hop in the hierarchy costs accuracy: stratum indicates the distance to the primary source. A stratum 0 is the atomic clock or the GNSS receiver; a stratum 1 is the server wired directly to it; a stratum 2 asks the stratum 1, and so on up to a practical maximum of 15 hops.
Correcting without breaking anything: slew versus step
When the skew is small, NTP does not move the clock at once: it drags it (slew). The Linux kernel offers the adjtimex() call for this, adjusting the clock frequency in parts per million over an interval until the error disappears. It is the difference between steering and slamming the brakes: a sudden backward jump can make two timestamps swap order, make make believe a file comes from the future, or invalidate a perfectly valid certificate.
Only at boot time, when the skew may be minutes or years, is a step accepted (or a makestep in chrony, limited by number of occurrences). And there is a case where neither technique is comfortable: leap seconds. Because the Earth’s rotation is not constant, Coordinated Universal Time has been adjusted by adding a second on 27 occasions since 1972. That extra second can repeat an instant already seen, and it has taken real systems down. That is why many providers apply leap smearing: instead of injecting the second abruptly, they spread the correction over hours. Google does it over 24 hours, Amazon over 24, and Meta over 18.
The software: from ntpd to chrony
The classic NTP daemon has been displaced on most distributions by chrony and systemd-timesyncd. timesyncd is minimalist: an SNTP client that synchronises once at boot and then at intervals, with no advanced filters. chrony implements a hybrid phase-and-frequency loop, able to estimate the crystal’s drift while disconnected and to converge in a handful of packets. On virtual machines it is especially useful, because the guest clock slips with every hypervisor pause: there is even the notion of a paravirtualised clock, such as kvm-clock, which reads the time from the host at almost no cost.
Security came late. For decades, NTP was unauthenticated UDP traffic, easy to spoof from anywhere on the network. NTS (Network Time Security, RFC 8915) remedies that: it first establishes a key with TLS 1.3 and then signs the time packets with authentication codes, so an attacker cannot shift a victim’s clock into the past to replay an expired token.
PTP: when the millisecond is not enough
For stock exchanges, smart power grids and 5G radio base stations, NTP falls short. The IEEE 1588 standard, known as PTP (Precision Time Protocol), attacks the root of the error: instead of measuring the instant the software sees the packet, it stamps the instant the network card receives it, using a hardware counter. That is hardware timestamping. With it, the offset no longer includes the time the operating system takes to service the interrupt, and you move from microseconds to tens of nanoseconds.
PTP also changes the topology. Instead of a hierarchy of servers, clocks elect the best candidate through the Best Master Clock algorithm (BMCA): each node announces its quality, its class, its estimated accuracy and its identifier, and all of them converge on the same winner. There are also transparent clocks: switches that do not merely forward the packet but write into it the time it spent queued, correcting the switch’s own jitter. ITU-T G.8275.1 adds a profile for telecom networks with chained clocks, and in 5G phase synchronisation between base stations is a requirement for antenna coordination to work.
The extreme case: asserting the time with uncertainty
Spanner, Google’s distributed database, turned the problem around. Instead of trying to make all clocks agree exactly, it assumes they do not and exposes an error bound: the TrueTime API returns an interval [earliest, latest] and the system waits as needed before confirming that a transaction has finished. In exchange for those milliseconds of waiting, it can order transactions globally without coordinating participants. The idea has caught on: AWS publishes an atomic clock service with certified uncertainty (ClockBound) and modern operating systems offer clocks with bounded error.
The lesson is the same as everywhere else in distributed computing: time is not a fact you read, it is a parameter you estimate. NTP, PTP and the cloud variants are nothing more than different ways of estimating that parameter accurately enough for the rest of the system to stop worrying about it.
