It Was Never the Model

A few days ago I published a post about one failure: I benchmarked a 27B model carefully on a single GH200, then upgraded the inference engine for an unrelated reason and got 43 percent more decode throughput and nearly four times the context budget from the identical configuration. The engine had been reserving a growing KV cache for all 64 layers when only 16 of them needed one.

The conclusion was that your engine is a lagging implementation of your model’s architecture, and that when it lags it usually does not crash. It just quietly hands you less than the architecture is offering.

Then I went back through my notes from the same stretch of evenings, and found six more.

Every one of them I had first written down as a finding about the model. Not one of them was.

”The model fails retrieval at 231k”

I ran a needle test: bury a specific value roughly 231,000 tokens deep in a prompt and ask the model to read it back. It failed. I recorded that long-context retrieval was broken and moved on, mildly disappointed in the model.

My probe capped output at 32 tokens. It is a thinking model.

The reasoning tokens ate the entire budget before the model ever emitted a character of answer. Raise the cap, or disable thinking for the probe, and it returns the value exactly, every time.

I came very close to filing a defect report against a model because of a number in my own test harness.

”This model is chatty and incompetent at tools”

Different day, different model, and this one appeared unable to use tools at all. The agent loop would stall. The model would ramble about what it was going to do instead of doing it. My note said it was not ready for agentic work.

The problem was a missing --tool-call-parser flag.

Models do not emit JSON tool calls. They emit text in whatever format they were trained on, and the server parses that text into the structured tool_calls field your agent framework is looking for. That parser is per-architecture and you select it with a flag. Pick none and the model’s perfectly well-formed tool call lands in content as a string, where your framework never looks.

The detail worth keeping is what passed while this was broken. The models endpoint: fine. A chat completion: fine. A plain prompt: fine. Every cheap check I had was green. Only running the actual agent loop, with a real tool it needed to call, surfaced it.

You are not talking to the model you think you are

I had a client config listing several models against one endpoint, which is a completely ordinary thing to do. It is also safe on one engine and quietly dangerous on another.

client config lists:  model-a, model-b, model-c
server actually has:  model-b
you request:          model-a
you receive:          model-b, with no error of any kind

One engine validates the model field in a request and returns a 404 on a mismatch. Another ignores the field entirely and answers with whatever it happens to have loaded.

I do not know how many of my early numbers were measurements of a model I was not trying to measure. I know it was more than zero, because I found the discrepancy by noticing a process start time that predated a config change.

The same flag means two different things

I set a 131,072-token context. Conversations started overflowing well before that, and the error mentioned context size, so my first instinct was that the model’s advertised window was optimistic.

On one engine that flag is per sequence, and concurrency is a separate setting. On the other it is the total KV budget, divided across parallel slots, and the default is four slots. So each conversation was getting 32,768 tokens.

Same flag name. Same units. Silently divided by four.

7,178 tokens to say “ok”

The prompt was, in full: Say exactly: ok

Unbounded, the reasoning budget defaults to unrestricted. Worse, without telling the server to separate thinking from output, those thousands of tokens land in the message content, get written into conversation history, and refill the context on every single turn. My client sat there auto-compacting forever.

The model looked incapable of a short answer. A harness setting had become a personality judgment.

The obvious optimization was 1.85x slower

This one is the most interesting, because the intuition is genuinely reasonable.

A mixture-of-experts model splits each layer into many expert sub-networks and routes each token to a handful of them. The total weights are enormous but only a slice is active per token, and there are far too many to fit in GPU memory. So you park them in the 480 GB of memory attached to the CPU.

At which point the obvious move is to let that CPU, which has 72 cores and owns the memory the weights are sitting in, compute those expert layers. No transfers required.

Letting the GPU reach across the interconnect and read those weights instead was 1.85x faster. The link is fast enough that remote reads beat local compute on the weaker processor.

I want to flag the limit on this one honestly, because it is the case most likely to be over-generalized. That result is a property of this machine, where CPU and GPU share a coherent, very wide link. On a conventional PCIe-attached card the same choice would likely invert. It is not a rule you can carry between machines, which is rather the point.

What they have in common

Seven failures. Every one of them looked like a model problem when I wrote it down, and none of them was.

They also share a shape. Not one of them threw an error. The model loaded. It answered correctly. It passed the checks I had. In four of the seven, the system reported success while doing the wrong thing, and in the other three it reported a real symptom that pointed at entirely the wrong cause.

That is what makes this class expensive. A model that refuses to load is a five-minute problem, because the failure names itself. A model that loads and serves plausible output while your harness quietly mis-specifies it can run for weeks. Mine did.

The other thing they share is that every one lived in configuration I had inherited rather than chosen. A default output cap. An absent flag. A client config that was reasonable on the engine I wrote it for. A context number that meant something different than I thought. Nothing here was exotic, and nothing here was the model.

What I actually changed

Test the loop, not the endpoint. The parser failure passed every cheap check I owned. The only thing that caught it was exercising the real agent loop against a real tool. If your system’s job is to call tools, a chat completion returning text is not evidence that it works.

Suspect the harness before the model, at least once. When something looks like a model defect, the cheapest possible next step is to ask what in my own setup could produce that exact symptom. It costs a minute and it would have saved me most of this list.

Write down what you held constant, and treat it as a claim. All seven of these live in things I set once and stopped thinking about. A default is an assertion that a value does not matter, and you almost never go back and check it.

Be careful which findings you generalize. The placement result is true on this machine and probably false on yours. Several of these are engine-specific rather than universal. A finding about a system you configured is not automatically a finding about the field.

I should be clear about scope, the same as last time. Everything here is throughput, retrieval, or observed behavior on one machine, worked on in evenings over a couple of weeks. I have run no quality evaluation of any kind, and nothing here says any model is better or worse than any other.

Which is the joke, really. I set out to evaluate models and spent a couple of weeks of evenings learning about my own tooling instead. Seven times I wrote down something a model had done wrong, and seven times the model had done nothing wrong at all.

If you run open-weight models on your own hardware, I would genuinely like to know which of these you have already hit. My guess is the parser one, and my guess is you blamed the model too.