AirLLM runs 70-billion-parameter models on a single 4GB card
An open-source tool called AirLLM lets enormous AI models run on ordinary consumer graphics cards by feeding them through the chip one slice at a time. It works because the thing everyone measured — total model size — was never the real limit.
There is a rule of thumb in machine learning that has hardened into something close to a law: to run a model, you need graphics memory roughly equal to the model's weight. A large language model with seven billion parameters — parameters being the adjustable numbers a model absorbs during training, each stored as a few bytes — wants something like fourteen gigabytes. Scale that up and the arithmetic gets ugly fast. DeepSeek-V3, with 671 billion parameters, would need roughly 2.7 terabytes at full precision. No graphics card you can buy for a home computer comes within a hundred times that.
Which makes it worth asking a narrower and more literal question than "how do we shrink these models?" At the precise moment a model is generating a word, what actually has to be sitting in memory?
A hundredfold in parameters, a doubling in memory
The gap did not open gradually. In 2023, a seven-billion-parameter model was considered large. Within a year the interesting open models were ten times that size. Now the frontier of publicly released weights sits at 671 billion for DeepSeek-V3 and 2.8 trillion for Kimi K3.
Consumer graphics processing units — the chips originally built to draw video game frames, which turned out to be excellent at the dense arithmetic neural networks require — did not follow. Their onboard memory, called VRAM, crept from 8GB to 12GB to 16GB over the same period. Parameters grew by a factor of a hundred; affordable memory roughly doubled.
That mismatch then did what mismatches do: it shaped everyone's assumptions. If the biggest models plainly could not fit on the biggest consumer card, then running them was a data-centre problem, and the tools, the tutorials and the pricing all got built for data centres. The constraint became the map. Very few people went back to check whether it had been measured correctly.
What a transformer is doing while it answers
The models in question share an architecture called a transformer, and a transformer is built as a stack of near-identical layers — 80 of them in a 70-billion-parameter Llama, more in the larger models. Each layer takes the output of the layer below it, performs its own set of multiplications, and hands the result up.
The important word is stack. Layer 40 cannot begin until layer 39 has finished, because layer 39's output is layer 40's input. There is no way to compute them simultaneously; the dependency is baked into the design. When your model is generating text, exactly one layer is doing arithmetic at any given instant. The other 79 are idle, holding weights that will not be touched for some fraction of a second, or have already been used and will not be needed again until the next word.
Standard practice loads all 80 into VRAM anyway and leaves them there. That is fast, and for a data centre with enough cards it is obviously correct. But it is a choice about convenience, not a requirement of the mathematics.
Load, compute, discard, repeat
AirLLM, an open-source project from developer Gavin Li, makes the other choice. Before the first run it splits a downloaded model into its individual layers and writes them to disk as separate files. Then, during generation, it moves one layer at a time onto the GPU, runs the computation, throws that layer out of VRAM and pulls in the next.
The memory the GPU needs is therefore not the size of the model. It is the size of its largest single layer, plus a little working space. And a single layer of a 671-billion-parameter model is about four gigabytes — three orders of magnitude smaller than the 2.7 terabytes the whole thing occupies.
This is the inversion, and it is worth stating plainly: the VRAM required to run a large language model does not scale with its total parameter count. It scales with its biggest slice. Nothing about the model has been reduced to achieve this. AirLLM does not shrink the weights the way model compression techniques do, and it does not prune or distil anything. The full model still has to be downloaded and still occupies its full size on disk — the disk is simply doing the storing while the GPU does the computing.
The published figures follow from that logic. A 70-billion-parameter Llama runs in about 4GB of VRAM. Llama 3.1 at 405 billion parameters runs in about 8GB. DeepSeek-V3 at 671 billion runs in about 12GB — a mid-range gaming card.
Why 2.8 trillion parameters fit in under four gigabytes
The largest models bend the numbers further, because they are built as a mixture of experts. Instead of one monolithic layer, each layer contains many smaller specialist sub-networks, and a routing mechanism sends each incoming token to only a handful of them. The rest of that layer sits out the calculation entirely.
If you are streaming pieces from disk anyway, you can go finer than a whole layer and fetch only the experts a token actually routes to. That is how AirLLM reports running Kimi K3 — 2.8 trillion parameters, the largest openly released model to date — in 3.72GB of VRAM, measured end to end on a single professional card. A model of 2.8 trillion parameters fitting in less memory than a 70-billion one is not a paradox once you accept that neither number was ever the relevant one.
The bill arrives in seconds per word
None of this is free, and the cost lands in exactly one place: time. A GPU reads from its own memory at rates measured in terabytes per second. It reads from an SSD at a small fraction of that. Every layer swap is a transfer across that slower link, and the model does not produce a single word until all of them have completed, once per token.
So this trade only makes sense where latency tolerance is high — where you want an answer eventually rather than immediately, and where the alternative is not getting one at all. It is a technique for people whose constraint is hardware they already own rather than time they are willing to wait.
There is also a firm boundary on where it applies. Layer streaming works for inference — running a finished model — because inference only ever moves forwards through the stack. Training moves forwards and then backwards, propagating error signals down through every layer, and that backward pass needs the layers and their intermediate results held together. Whether anything similar can be made to work for training remains an open question. For now, the discovery is narrower and still substantial: the wall was in the wrong place, and it was in the wrong place for years.
Questions
Does AirLLM reduce the quality of the model's answers?
Not by default. The weights are used at their original precision — nothing is quantised, pruned or distilled. The model computes exactly what it would compute on a large GPU; it just does so one slice at a time.
Do I still need to download the whole model?
Yes. The full model has to be downloaded and then split into per-layer files on disk, which is disk-intensive. What drops dramatically is the graphics memory required, not the storage required.
How slow is it in practice?
The project does not publish a single figure, but the bottleneck is clear: every layer must be transferred from disk to the GPU for every token generated, and disk bandwidth is a small fraction of GPU memory bandwidth. It suits work where waiting is acceptable.
Can this technique be used to train models too?
No. Training requires backpropagation, which passes error signals backwards through every layer and needs them available together. Layer streaming only works for inference, where computation flows in one direction.