One GH200, sitting in a rack

One address space across two memory tiers. For everything in this talk the workload never leaves the left half, so you can forget the Grace side exists. This is a story about software.

Every request runs in two completely different phases

Five slides of foundations, about four minutes. If you serve models for a living, this is review and you can rest. If you mostly call APIs, this is the part that makes everything after it land. I would rather over-explain than have half the room politely lost.

Almost everything surprising about serving a model lives in the gap between those two phases. Hold on to this, the whole talk hangs off it.

To produce one token, the GPU reads the entire model

Not part of it. All of it. Every weight, out of memory, for every single token. Then it does it again for the next one.

So single-stream generation is not limited by how fast the GPU can do maths. It is limited by how fast it can move bytes. The tensor cores, the expensive part you actually bought the card for, sit mostly idle.

Prefill is the opposite: one pass over a huge batch of tokens, math units saturated. Same silicon, opposite bottleneck.

Why context costs you memory

When the model generates token 5,000, attention makes it look back at all 4,999 tokens before it. Recomputing those every step would be brutally quadratic.

So the server keeps a KV cache: for every token that has passed through, it stores that token's keys and values, once, and reuses them forever after.

The win is that each token is processed once rather than once per subsequent step. The cost is that the cache grows with every token. When someone says a model "supports 262k context," what they mean is: if you can afford the cache.

So what is actually in that cache?

Attention works by each token asking a question. That question is a query. Every earlier token offers a key, saying roughly what it is about, and a value, carrying its actual content. Models run a couple of dozen of these comparisons in parallel, and each parallel copy is called a head. Only the keys and values get cached, and they do not have to be one set per head.

That is grouped-query attention, and it is the only reason a 262k window fits on this box at all. Your context length is an architecture decision, not a hardware capability.

The two things everyone tunes

Quantization stores each weight in fewer bits. FP8 instead of BF16 halves the bytes, and since decode is bandwidth bound, fewer bytes per weight is directly fewer bytes to read per token. Speculative decoding attacks the same bottleneck from the other end:

These are the two knobs I spent that night measuring. Measuring them carefully is exactly why I thought I understood my setup.

I did the disciplined thing

A 27B model. One variable at a time. Repeat runs to confirm every number.

• Speculative decoding on and off: 63.1 to 112.0 tok/s, a 1.78x gain for one flag
• FP8 against BF16, with the draft head active on both
• Separated prefill from decode by streaming and timing the first content chunk
• Every configuration confirmed by a second identical run

Clean, repeatable, defensible. I wrote it all down and I was pleased with it.

I upgraded the engine for an unrelated reason

Different model I wanted to try. Its architecture was not registered in the version I had. So I built a second environment alongside the first and, while it was sitting there, re-ran the identical benchmark on it.

Same model. Same quantization. Same speculative config. Same memory utilization. Same hardware. Same harness script.

The variable I never tested moved more than everything I did.

The most important number is the one that did not move

Prefill was 6,266 tok/s before and 6,249 after. Unchanged. That is not a footnote, it is the whole attribution.

If both had moved I would be hunting a config mistake in my own setup. Because only the bandwidth-bound half moved, I can point at the serving stack. A null result did the attribution.

A config field I had never once looked at

"full_attention_interval": 4

This model is not a uniform stack. One layer in four is ordinary full attention; the other three are not.

This is the second way the architecture cuts your context bill. The first was sharing key and value sets across heads; this one skips three quarters of the layers entirely. The older engine did not know that. The ratio predicts 4x; I measured 3.75x. Inferred from release notes and that ratio, not from per-layer cache logs.

Nothing errored. Nothing warned.

• It answered correctly
• It passed needle retrieval at 231,000 tokens
• It drove a working agent loop for weeks

I just had three quarters less context than I thought I did.

This is the shape of the whole talk. The loud failure is the easy one: the model refuses to load, you go find a newer engine, you move on with your day. The quiet failure is the expensive one.

I had already checked this

2 (K and V) x 4 kv_heads x 256 head_dim x 64 layers = 128 KiB / token
54.82 GiB / 128 KiB  =  ~449,000 tokens
engine reported       =   448,448 tokens     match

It matched to within a rounding error, so I moved on satisfied.

The arithmetic was correct. The premise was wrong.

I confirmed the engine was doing exactly what it claimed to be doing, and quietly mistook that for confirming it was doing the right thing. Those are different claims, and only one of them was worth my time. We were wrong together, in perfect agreement, for months.

Then I went back through my notes.

The engine version was not the only thing that had fooled me. It was the first.

Six more from the same rack, same stretch of evenings. Each one was first written down as a finding about a model.

"The model fails retrieval at 231k"

I recorded that long-context retrieval was broken: the model could not find a value buried 231,000 tokens deep. It is a thinking model, and my probe capped output at 32 tokens.

I nearly filed a defect report against a model because of a number in my own harness.

"This model is chatty and incompetent at tools"

Every cheap check passed while this was broken. Models endpoint: fine. Chat completion: fine. Plain prompt: fine. Only running the actual agent loop caught it.

You are not talking to the model you think you are

One engine validates the model field in a request and 404s on a mismatch. Another ignores it entirely and answers with whatever it happens to have loaded.

client config lists:  model-a, model-b, model-c
server actually has:  model-b
you request:          model-a
you receive:          model-b, with no error of any kind

A multi-model client config is completely safe on one and actively dangerous on the other. Every benchmark you ran against "model-a" is a benchmark of model-b.

The same flag means two different things

You find out when the client overflows, and the error talks about context size, so you go and blame the model's context window.

7,178 tokens to say "ok"

The prompt was: Say exactly: ok

Unbounded, those thoughts land in the message content, get stored in history, and refill the context every single turn. The client auto-compacts forever and the model looks incapable of a short answer.

The obvious optimization was 1.85x slower

A mixture-of-experts model splits each layer into hundreds of "expert" sub-networks and routes each token to just a handful, so total weights are enormous but only a slice is active per token. Too many to fit in GPU memory, so you park them in the 480 GB attached to the CPU. The intuitive move is then to let that CPU compute the experts whose weights sit in its own memory. No transfers, right?

The interconnect beats the weaker processor, decisively. And this would likely invert on a PCIe-attached card, which is exactly the point: it is a property of the machine, not a rule you can carry between machines.

Every one was first filed against a model

• looked like the model fails long-context retrieval  was my output cap
• looked like the model is bad at tools  was a missing parser flag
• looked like model A's numbers  was model B answering silently
• looked like a small context window  was a flag with two meanings
• looked like the model cannot be terse  was an unbounded reasoning budget
• looked like a memory-placement law  was one machine's topology
• looked like the model's context budget  was the engine not knowing its architecture

Three of the first four would have been written down as findings about the model rather than about the harness.

Your harness is a lagging implementation of your model's architecture.

Architectures are changing structurally, not just getting bigger. Every new mechanism has to be implemented in your serving stack before you get anything like what it offers, and that implementation lands later.

What I actually changed

• Engine version goes on the benchmark axis, not in the environment notes. I record it the way I record quantization, because it behaves like a variable.
• Test the full loop, not the endpoint. Every cheap check passed while tool calling was silently broken. Only the real agent loop caught it.
• One variable at a time, even when it feels slow. Two knobs I tuned partially cancel; flipping both together would have hidden that entirely.
• Re-test your tuning peaks when either the model or the engine moves. Both shifted my optimum by more than 40 percent.
• Checking that a system does what it says is not checking that it is right. That is the one that cost me the most.

Everything you hold fixed is a claim you are making.

A control is an assertion that something does not matter. You rarely go back and check it, because checking it is the same work as testing it, and you already decided it was background.

Somewhere in your stack there is a version number, a default, or an inherited flag you have stopped seeing.

The background of your benchmark is someone else's independent variable.