Take a photo of a white wall. The sensor behind the lens has tens of millions of points spread over a grid a few millimetres wide, and every one of them has done exactly the same thing: count how many photons landed on it. Not one of them knows what colour the light was. Everything the photograph knows about colour was added afterwards, with mathematics, and even so the result looks true.
Every point on the sensor only measures how much light arrives
A photosite is essentially a well dug into silicon. Silicon has a band gap of 1.12 electron-volts, so any photon with enough energy — that is, with a wavelength below about 1,100 nanometres — can knock an electron loose inside the material. One photon absorbed, one free electron. The electrons pile up in the well during the exposure and are counted at the end. That count is the signal, and there is nothing else.
The efficiency of the process is called quantum efficiency: in a good sensor, between 60% and 80% of the photons that arrive end up converted into a useful electron. The well has a limit, the full-well capacity, which in a phone pixel is a few thousand electrons; when it fills up, that point reads its maximum and appears as pure white in the photo, no matter how much more light comes in.
The most important noise does not come from the design but from physics: photons arrive at random. If N arrive on average, the typical variation is the square root of N. That means doubling the light doubles the signal but increases the noise by only 41%, and that to gain a factor of two in signal-to-noise ratio you have to quadruple the light. That is why your phone’s night mode does not increase the sensitivity of anything: it takes between eight and twelve photos in a row and adds them together.
From electrons to a number
When the exposure ends, the charge is turned into a voltage and an analogue-to-digital converter writes it down as an integer of 12 or 14 bits, that is, one of 4,096 or 16,384 possible levels. In today’s CMOS sensors there is not a single converter for the whole chip: there is one per column, working in parallel, which is what makes it possible to read millions of points in thousandths of a second.
How they are read matters. A rolling electronic shutter sweeps the rows one after another, so the first and the last row of the image were not captured at the same instant: that is why the blades of a fan come out bent. A global shutter, which exposes every row at once, requires storing the charge in a memory inside each pixel, and that takes up useful surface. The solution of recent years is to stack two wafers: the one with the photosites on top and the logic wafer right underneath, joined by microcontacts.
Colour is not in the silicon: it is in the filter
Here is the catch mentioned at the start. Silicon responds to a red photon exactly as it does to a blue one: in both cases, by counting one. The energy of the photon is not measured. To recover colour, a microscopic filter is placed in front of every point, and the filters are arranged in the so-called Bayer pattern: blocks of four, with two green filters, one red and one blue.
Green takes half the points for two reasons at once: the human eye is far more sensitive in that part of the spectrum, and the green channel is a good approximation of luminance, that is, of the brightness we perceive. Besides the filter, each point carries a microlens that concentrates light that would otherwise fall on the dead space between transistors, and the whole assembly is covered with a filter that cuts infrared, because silicon detects it and the eye does not.
The result is uncomfortable: each point on the sensor measures a single channel. It has its green, or red, or blue level, but not all three. Put another way, two out of every three numbers that make up the colour of each pixel in your photo do not exist and have to be invented.
Inventing two out of every three numbers
That invention is called demosaicing, and it is the image processor’s first serious job. The naive version interpolates each channel from the average of its neighbours, which works in smooth areas and produces disasters at edges: false colours where there is a black line, and rainbow patterns on fabrics and fine grids, because the filter pattern interferes with the pattern in the scene. Modern algorithms look at the direction of edges and rebuild each channel along the direction in which colour changes least, then correct the zigzag artefacts the process leaves behind.
The industry has found a physical shortcut: the Quad-Bayer sensor groups four photosites with the same colour filter, so the pattern repeats in 2×2 blocks. It can be read as a 50-megapixel sensor, reconstructing the Bayer pattern in software, or each group of four can be added together as if it were one large point, which yields 12.5 megapixels that each gathered four times more light. That is the trick that lets a phone advertise 50 megapixels and shoot at 12 in practice.
Noise, dynamic range and the ISO misunderstanding
The signal-to-noise ratio of a point depends on two figures: how many electrons fit in its well and how much noise the electronics adds when reading them. That second term, read noise, is a matter of a few electrons and can be measured today. The distance between the two, expressed in exposure steps (each step is twice the light), is the dynamic range: about twelve steps on a phone, twenty or more on a large camera with huge photosites.
One misunderstanding is worth clearing up: raising the ISO does not make the sensor more sensitive, because the probability of capturing a photon is a physical property that does not change. What it does is amplify the voltage before digitising it, and with it the noise that was already there. The exception is dual-gain sensors, which switch between two conversion capacitors: a large one for bright scenes and more range, and a small one for low light, where a single electron produces a larger voltage.
From the number to the photo you see
What comes out of the sensor is not yet a photograph. The colour of the light that lit the scene has to be estimated and corrected — that is white balance —, the sensor’s three channels have to be translated into the display’s colour space through a matrix, a tone curve has to compress the extremes, and a decision has to be made about how much noise to smooth away at the cost of losing fine detail. Almost all of that happens in a fraction of a second, quietly, without the photographer ever noticing.
And then there is computational photography, which is where it comes closest to magic. In a good number of phones, every photo is really an aligned burst: adding shots separated by a shift smaller than one pixel gains light and fills in information that no single frame had. The hard part is not the addition but detecting what moved between frames, so that a pedestrian is not blended into the background.
Sensors that count photons one by one
The frontier is no longer about capturing light better but about capturing it differently. Single-photon avalanche diodes (SPADs) can detect an individual photon and record the exact instant it arrived, which gives millimetre-accurate distance measurements and photon counts with no read noise. Event sensors go further: they do not produce frames, but each point sends a notification when the brightness it sees changes, with microsecond resolution and minimal power consumption. For the machine vision of a robot or a car, that stream of events is more useful than twenty-five frames per second.
In the end, a camera does not see: it counts. Colour is a statistical reconstruction built from a filter and from what the neighbouring points measured, and the quality of a photograph depends on three things that can be measured separately: how much light reached the sensor, how much was lost along the way, and how much computing power is spent afterwards fixing what was missing.
