Lower Precision Lets Cloudflare Double AI Requests Per GPU

A large language model is not a program that runs on your laptop. It is an enormous pile of numbers sitting on graphics chips in a data centre, and while you are typing at it, thousands of strangers are typing at the same copy. That sharing is what makes the service cheap enough to offer. It is also what limits it: each chip has a fixed amount of fast memory, and once that memory is full, the next person in the queue waits.

Which sets up a temptation and a worry. The temptation is to hold everything on the chip in a coarser form — fewer digits per number — so that more fits. The worry is that a model assembled out of blurrier numbers will produce blurrier answers. Cloudflare, the American company best known for keeping websites online and fending off attacks against them, also rents out AI capacity from those same data centres, and has published an unusually specific account of doing exactly this to two of the heaviest models it hosts: Kimi, from the Chinese lab Moonshot AI, and GLM, from Z.ai. Both are mixture-of-experts designs, meaning only a fraction of their parameters is consulted for any given word — cheaper to calculate with, but no smaller to store. Both are also built to read very long documents, and that turns out to be where the trouble starts.

The running notes fill the chip before the model does

To write text one word at a time without re-reading the whole conversation for each new word, a model keeps running notes on everything it has processed so far. Engineers call this the KV cache, after the "keys" and "values" that the model's attention machinery produces for each chunk of text and then keeps on hand. The longer the input, the fatter those notes become. For a model designed to swallow a novel's worth of context, it is normally the notes, rather than the model's own parameters, that exhaust the chip's memory first.

That produces the choice at the heart of all of this. When memory is full, an operator can either refuse the next request, or find a way to hold the same information in less space.

Half the digits, twice the customers

Cloudflare's first move was to store the notes at half the usual precision. The default is sixteen bits per number; the company switched to an eight-bit floating-point format, which halves the space each entry takes. On Kimi K2.6, the memory freed up raised the amount of context a chip could hold from roughly 686,000 tokens to about 1.37 million — a token being, very roughly, three-quarters of a word.

The counterintuitive part is that this does not make anything faster. It makes each word marginally slower, because the chip has to unpack the compressed values as it reads them. Compare the two setups at any fixed number of simultaneous conversations and the sixteen-bit version wins by a few percent. The benefit shows up somewhere else entirely: in how many conversations the chip can hold at once. On Nvidia H200 chips, the sixteen-bit configuration hit a wall at 32 simultaneous requests and had nowhere to put a 33rd. The eight-bit version carried on to 64, where it was generating 2,192 tokens per second — about 41 per cent more than the best the sixteen-bit setup ever reached, at roughly 30 per cent lower cost per token.

None of that would be worth having if the answers changed, so the company ran both versions through its evaluation suite. On the grade-school maths test GSM8K, the two scored 94.24 and 94.09. On MMLU, a broad general-knowledge exam, 89.11 and 89.04. Validity of the structured "tool calls" a model emits when it wants to use software came out at 92.2 and 92.6 per cent. The gaps run in both directions and sit inside the noise.

Decode and prefill want opposite things

The notes are one claim on memory; the model's own weights are the other. For GLM 5.2, Cloudflare squeezed those weights from eight-bit floating point down to four-bit integers. The stored model shrank from 705GB to 421GB, about 40 per cent. Spread across an eight-chip deployment, each chip's share fell from roughly 88GB to about 52GB, leaving space for something like 1.18 million tokens of cache. Across every benchmark the company runs, accuracy stayed within 0.8 points of the uncompressed version.

Smaller weights also make the model faster at writing, and the reason exposes something structural about how running a model works. There are two phases. Reading the prompt — prefill — is a burst of arithmetic that the chip's calculating power limits. Writing each subsequent word — decode — requires streaming the weights out of memory once per word, so the ceiling is how fast data moves, not how fast it multiplies. Shrink the weights and every word arrives sooner. Serving a single request, GLM went from 60 tokens per second to 92, a 55 per cent gain. At 64 concurrent requests, 1,672 became 1,933.

Prefill moves the other way. Four-bit numbers must be expanded again before the model can multiply with them, and that unpacking costs time: prefill throughput fell from about 10,160 tokens per second to 8,660. Because Cloudflare runs the reading and writing phases on separate pools of chips, this becomes a choice rather than a compromise. The decode pool gets the four-bit weights and the eight-bit cache; the prefill pool keeps the larger, faster versions of both.

One in a billion happens often at this volume

Both techniques do the same underlying thing: they pack far more simultaneous requests onto a single chip. That is the point, but it has a consequence. Hundreds of requests are now reading and writing pages of one physical block of memory, handed out and reclaimed as conversations begin and end. The mechanisms that make this efficient — chopping the cache into pages, continuously reshuffling which requests are batched together, reusing pages whose contents recur — all rest on the bookkeeping being exactly right. At the volumes Cloudflare handles, a mix-up with a one-in-a-billion chance is not a hypothetical; it happens on a schedule. And the failure it produces is one user's text appearing inside another user's answer.

So the company added a check. Each physical page carries a tag that changes whenever the page is handed to someone new, and the server keeps a record of which pages and tags each request ought to be touching. Before supported read operations during decode, the record is compared against reality. If it does not line up, that request is killed rather than allowed to return whatever was in the page.

Whether a safeguard like that survives contact with production depends entirely on its price. Measured on a mid-sized model in a two-reader, two-writer configuration with 8,192-token inputs and 1,000-token outputs, throughput dropped by less than 1 per cent and worst-case latency rose by less than 1 per cent. It stayed cheap because the validation runs as its own pass rather than being folded into the attention calculation, where it would have created a race between groups of threads on the chip. It is switched on per deployment; anything that does not want it gets a do-nothing version that costs nothing measurable.

What connects the three moves is that none of them is a general-purpose speed-up. Each is aimed at a different bottleneck: coarser notes where memory is the constraint, smaller weights where data movement is, the untouched full-precision path where raw arithmetic is, and a cheap consistency check to cover the crowding that the first two create.

Whether the same recipe transfers is the open question. Every figure above belongs to a particular model on a particular chip. Cloudflare says it is now extending eight-bit caches across more of its fleet, testing a four-bit format on Nvidia's newer Blackwell chips, and working towards leaving the integrity checks on everywhere. Each of those is another pairing of model and hardware whose behaviour has to be measured rather than assumed.