A 304-Billion-Parameter Model Now Runs on One AMD GPU

The arithmetic that makes one card plausible

The usual way to serve a frontier-scale language model is to buy several accelerators and split the model across them. The weights are too large for one chip, so they are sharded, and the machines spend part of their time shuffling intermediate results between cards. That is the assumption baked into most deployment guides. The question worth asking is whether it still holds — whether a 304-billion-parameter model can be served, in production, from a single GPU, at speeds a real user would tolerate.

One engineer has now published the configuration that answers it: a repository of patches, pinned versions and tuning tables for running DeepSeek V4 Flash on one AMD MI300X.

Start with the memory, because everything else follows from it. A GPU holds the model's weights in high-bandwidth memory — HBM, the fast memory sitting next to the processor — and it must hold all of them at once, plus a working scratchpad for every conversation currently in flight. The MI300X carries 192 GB of HBM, roughly two and a half times what an Nvidia H100 offers. DeepSeek V4 Flash, loaded exactly as its makers shipped it, occupies 156.67 GiB.

That leaves about 35 GB. It is enough, but only just, and only because of how the model is built. It is a mixture-of-experts design: rather than running all 304 billion parameters for every word, it routes each token through a small subset of specialised sub-networks. The experts ship in a compact four-bit numeric format, which is why the download is 156 GB rather than several times that. Nothing was compressed further to make it fit. Engineers who have worked through AMD's wider product line note that this native compactness also puts the model within reach of the MI350P — a card that plugs into a standard server slot rather than the specialised module the MI300X uses, and which has less memory at 144 GB, but enough of it.

Where the numbers stop being right

Fitting is not the same as working. The first thing that broke was not speed but correctness, and the cause was a disagreement about what a byte means.

Modern inference leans on eight-bit floating point, or FP8 — a way of storing numbers in one byte instead of four, trading precision for memory and bandwidth. There is an industry standard for it. The MI300X, whose chip generation predates that standard's adoption at AMD, implements a slightly different variant with different rules for how the exponent and the scale relate. Later AMD cards use the standard form. Software written against the standard and pointed at an MI300X does not crash; it reads the bytes, interprets them under the wrong convention, and can come out wrong by a factor of two in the scaling. Wrong by two, silently, is worse than a crash.

The official recipe for serving this model in vLLM — the open-source engine most people use for this job — covers Nvidia hardware and AMD's newer cards. It does not cover a single MI300X with this particular model release. So the gaps had to be filled by hand.

The repository's fixes are blunt in form and precise in content: full replacement copies of about nine Python files, mounted read-only over the versions inside the software container, each accompanied by a diff showing exactly what changed from the original. One rewrites the cache writer to emit AMD's FP8 variant in the memory layout the AMD kernel library actually expects, while leaving the standard path untouched for newer cards. Another fixes the expert-routing matrix, where padding lanes were being checked against the wrong boundary; under load, that corruption surfaced as the model inventing near-miss names for the tools it was supposed to call, and losing track of instructions in long prompts. Others handle larger memory offsets for big caches, enforce deterministic behaviour so identical prompts produce identical tool calls, and add a synchronisation fence between the CPU and the GPU that is described in a public bug report whose proposed fix was never merged.

Twenty-one matrix shapes and a cache in two tiers

With the output correct, the tuning began. AMD's kernel library ships lookup tables telling it how to organise each matrix multiplication for a given shape of data. Twenty-one of the shapes this model repeatedly asks for were missing from those tables, so the library fell back on generic choices. Measuring and adding them lifted single-stream decoding by 42 to 62 per cent, and by 10 to 35 per cent under heavier load. A geometry override for the expert kernels — the stock settings degraded sharply past a certain batch size — took one decoding path from 34.5 to 56.6 tokens per second.

The remaining memory was split in two. Twenty gigabytes of HBM holds the live cache of each conversation's context, with 96 GiB of ordinary system RAM behind it as an overflow tier for entries pushed out of the fast pool. The headroom is genuinely thin: warmed up, the card sits at 204.5 GB of the 205.8 GB it reports, and asking for a 30 GB cache instead of 20 fails outright during startup.

What eight people at once actually get

A single user sees around 168.6 tokens per second — comfortably faster than anyone reads. Eight simultaneous users get 542 tokens per second in aggregate, a median of 90.3 each. A burst of 64 streams sustains 830 tokens per second without running out of memory or throwing engine errors. Reading in a fresh prompt runs at roughly 7,000 tokens per second in the production settings, deliberately throttled so that one long document cannot stall everyone else; that choice cuts the wait for a short request queued behind a 52,000-token prompt from 8.2 seconds to half a second.

The concession is context length. The architecture supports a million tokens of working memory; what fits here, once the weights are loaded, is 256,000. That is a large window by any practical measure, but it is a quarter of what the model can do.

What this amounts to is a shift in where the boundary sits. Serving a frontier model no longer necessarily requires a multi-GPU machine. It can require one card with unusually generous memory, plus someone willing to chase down the places where a vendor's software has not yet caught up with that vendor's own silicon — without compressing the model to get there. Cloud providers now rent MI300X time by the hour, so the experiment is available to anyone who wants to repeat it, and a team that did the equivalent work across two of the cards has published its own findings.

Which leaves the harder question. The fixes here are specific to one model, one card generation and one nightly build of the serving engine. Whether that hand-work is a permanent tax on every new pairing of frontier model and accelerator, or a temporary gap the vendor stacks will eventually absorb, is not yet clear.