It Was Never the Model

Here is the system this post is about. A laptop running agent clients. A single GH200 in another room. Between them, a semantic router whose job was supposed to be picking which model answers each request: the small local one for trivial turns, the 27B on the big card for everything else. That premise did not survive measurement, and what the router turned out to be good for is a different thing than what I built it for. Getting there took eleven failures. Every one of them looked, at first, like a fact about a model. None of them was.

Everything the router does today, on one card. Two client dialects come in on the left: Claude Code speaks the Anthropic messages API, OpenCode and Codex speak OpenAI chat. The router reads four signals and makes three decisions. Three of the four signals pick a tier; the fourth strips tools. On the right, one GH200 runs the 27B, a speech model, and a guard model, in 78 of its 95.6 GB. The dashed lines are the honest part. Speech is not behind the router at all, because the router has no audio endpoint. The guard model is drawn dashed because it is measured and not wired, and a diagram should not claim more than the system does.

The box, briefly

Enough of the hardware to follow what went wrong. The GH200 has 95.6 GB of HBM3 on the GPU and 573 GB of LPDDR5X on the Grace CPU, joined by NVLink-C2C at 447 GB/s across 10 links. That link is wide enough that reading across it beats computing locally on the weaker side, which matters later.

One card, two memories, one address space. This is the box from part one, and everything below runs on it.

Prefill, the part where the model reads your prompt, is compute-bound and enormously faster on the card: about 6,900 tokens per second against roughly 290 on the laptop. Decode, the part that writes the answer, is bound by memory bandwidth, one token at a time, and everything about context length is really a question of where the KV cache for that context lives and how big the output budget is. Most of what follows is one of those two things, set by a default I never looked at.

Seven failures, none of them the model

A few days ago I published a post about one failure: I benchmarked the 27B carefully on this card, then upgraded the inference engine for an unrelated reason and got 43 percent more decode throughput and nearly four times the context budget from the identical configuration. The engine had been reserving a growing KV cache for all 64 layers when only 16 of them needed one.

The conclusion was that your engine is a lagging implementation of your model’s architecture, and that when it lags it usually does not crash. It just quietly hands you less than the architecture is offering.

Then I went back through my notes from the same stretch of evenings, and found six more.

Every one of them I had first written down as a finding about the model. Not one of them was.

”The model fails retrieval at 231k”

I ran a needle test: bury a specific value roughly 231,000 tokens deep in a prompt and ask the model to read it back. It failed. I recorded that long-context retrieval was broken and moved on, mildly disappointed in the model.

My probe capped output at 32 tokens. It is a thinking model.

The reasoning tokens ate the entire budget before the model ever emitted a character of answer. Raise the cap, or disable thinking for the probe, and it returns the value exactly, every time.

I came very close to filing a defect report against a model because of a number in my own test harness.

”This model is chatty and incompetent at tools”

Different day, different model, and this one appeared unable to use tools at all. The agent loop would stall. The model would ramble about what it was going to do instead of doing it. My note said it was not ready for agentic work.

The problem was a missing --tool-call-parser flag.

Models do not emit JSON tool calls. They emit text in whatever format they were trained on, and the server parses that text into the structured tool_calls field your agent framework is looking for. That parser is per-architecture and you select it with a flag. Pick none and the model’s perfectly well-formed tool call lands in content as a string, where your framework never looks.

The detail worth keeping is what passed while this was broken. The models endpoint: fine. A chat completion: fine. A plain prompt: fine. Every cheap check I had was green. Only running the actual agent loop, with a real tool it needed to call, surfaced it.

You are not talking to the model you think you are

I had a client config listing several models against one endpoint, which is a completely ordinary thing to do. It is also safe on one engine and quietly dangerous on another.

client config lists:  model-a, model-b, model-c
server actually has:  model-b
you request:          model-a
you receive:          model-b, with no error of any kind

One engine validates the model field in a request and returns a 404 on a mismatch. Another ignores the field entirely and answers with whatever it happens to have loaded.

I do not know how many of my early numbers were measurements of a model I was not trying to measure. I know it was more than zero, because I found the discrepancy by noticing a process start time that predated a config change.

The same flag means two different things

I set a 131,072-token context. Conversations started overflowing well before that, and the error mentioned context size, so my first instinct was that the model’s advertised window was optimistic.

On one engine that flag is per sequence, and concurrency is a separate setting. On the other it is the total KV budget, divided across parallel slots, and the default is four slots. So each conversation was getting 32,768 tokens.

Same flag name. Same units. Silently divided by four.

7,178 tokens to say “ok”

The prompt was, in full: Say exactly: ok

Unbounded, the reasoning budget defaults to unrestricted. Worse, without telling the server to separate thinking from output, those thousands of tokens land in the message content, get written into conversation history, and refill the context on every single turn. My client sat there auto-compacting forever.

The model looked incapable of a short answer. A harness setting had become a personality judgment.

The obvious optimization was 1.85x slower

This one is the most interesting, because the intuition is genuinely reasonable.

A mixture-of-experts model splits each layer into many expert sub-networks and routes each token to a handful of them. The total weights are enormous but only a slice is active per token, and there are far too many to fit in GPU memory. So you park them in the 480 GB of memory attached to the CPU.

At which point the obvious move is to let that CPU, which has 72 cores and owns the memory the weights are sitting in, compute those expert layers. No transfers required.

Letting the GPU reach across the interconnect and read those weights instead was 1.85x faster. The link is fast enough that remote reads beat local compute on the weaker processor.

I want to flag the limit on this one honestly, because it is the case most likely to be over-generalized. That result is a property of this machine, where CPU and GPU share a coherent, very wide link. On a conventional PCIe-attached card the same choice would likely invert. It is not a rule you can carry between machines, which is rather the point.

Then I caught myself, and two of them were mine

I wrote all of that down, felt appropriately humbled, and then went and built the router in the diagram: something to sit on my laptop, answer trivial requests with a local model, and forward the hard ones to the GH200. Reasonable premise. It did not survive measurement either.

The laptop only wins below roughly 180 prompt tokens. At 290 tokens per second of prefill against the GH200’s 6,900, the crossover arrives almost immediately, and every agent client carries a multi-thousand-token system prompt before the user has typed anything. Their traffic never lands inside that envelope. One real captured OpenCode request took 157 seconds on the laptop and 3.0 seconds on the GH200.

Three more failures came out of that work. The first has the same shape as everything above, somebody else’s default quietly doing something other than what it said. The other two are worse, because the thing that lied to me was my own measurement.

The log said reasoning was off

My decision config said use_reasoning: false. The router agreed with me, in writing, on every single request:

Reasoning mode disabled for model: local-fast

It then spent 549, then 1,700, then 2,486 completion tokens answering the prompt say ok.

That setting resolves to chat_template_kwargs: {enable_thinking: false}, which is the correct field. The problem is that Ollama’s OpenAI-compatible endpoint accepts it, ignores it, and says nothing. The control that actually works is a think parameter that exists only on its native /api/chat. Qwen’s /no_think token was ignored on that path too.

Same model, same prompt, two endpoints:

Case five in this post was a harness setting that made a model look incapable of a short answer. This is the same failure with an extra insult on top: the harness told me it had applied the fix. A silent harness is expensive. A harness that logs the opposite of what it did is worse, because it spends your attention defending the one thing you should be suspicious of.

Two quote characters chose the model

opencode run wraps your prompt in literal double quotes before it goes on the wire. Nothing else about the request changes. That alone moved my complexity classifier’s margin:

bare string          margin = -0.155
wrapped in quotes    margin = -0.085   crosses the routing boundary

The entire observed dynamic range of that margin is about 0.2. Two characters moved it by roughly a third of the usable scale, far enough to send an identical request to a different model.

The part worth sitting with is not the classifier. It is that I had been scoring the router against bare strings, a wire format no real client sends. I had built a careful evaluation of a situation that does not occur. Measured across both formats, my reported accuracy fell from 90 percent to an honest 75.

Your harness is not a transport. It is an input.

My cache-busting was cached

Simple question: which prefills a 49,000-token agent request faster, the laptop or the GH200? I measured 6.0 seconds against 6.5. Near parity. I wrote it down, and it was interesting enough that I nearly published it.

It was wrong. I had been salting the end of the last user message to defeat caching, which means the 49,000-token prefix in front of it was byte-identical on every run. Both backends were serving it straight out of prefix cache. I had measured the cache, not the machine.

True cold prefill is 169.6 seconds on the laptop against 7.1 on the GH200. Not parity. A 24x gap, pointing the opposite way from my first number.

Suffix cache-busting cannot invalidate a prefix cache. That is obvious once stated, and I still got it wrong, and it is the second time in this post that a caching layer made a benchmark say something false.

The router’s original premise did not survive any of this. What it turned out to be genuinely good for was protocol work, letting a client speak one API dialect to a backend that speaks another. Tier selection, the thing I built it for, was the smaller half.

What actually earns a tier

Having built the router for tier selection, I owe an account of what a tier has to do to deserve one. The answer turned out to be narrower than I expected: it has to do something the generalist genuinely cannot, not merely be better at something.

I added a vision tier, an 8B model with a 32k context, and routed anything carrying an image to it. Then I probed the 27B with rendered images and found it was already multimodal, with a 262k context. The vision tier was a downgrade in both dimensions. It was measured, found wanting, and removed the same day; the memory it freed is what the guard model now runs in.

A bigger model has the same problem from the other side. The 304B mixture-of-experts I tried is smarter than the 27B and 2.3x slower to decode, 73.4 tokens per second against 169.9. And when two models share the card, the faster one loses more: under contention the speculative-decoding 27B dropped 66 percent of its throughput and the plain version 52, though the fast one still won on absolute numbers, 74.2 against 43.4. A tier earns routing only when it does something the default cannot, and on this card that turned out to be a short list.

Claude Code on my GPU

The half of the router that survived is protocol translation. Claude Code speaks the Anthropic messages API. My backend speaks OpenAI chat. The router sits between them and lets each keep its dialect, which is the difference between a client that works and one that does not.

It was not free. Three small router patches were needed before Claude Code would complete a request, each one a field the router rejected that the real Anthropic API accepts. And on the way I found a genuine bug in vLLM’s Anthropic-compatible endpoint, where the final streaming event omits a field the schema requires. That is its own story, and a shorter one than this.

A calibration that measured nothing, confidently

The router picked up one more job after that: a prompt-injection guard. A signal scores each request, and anything that looks like an injection gets quarantined before it can carry tools.

Quarantine here does not mean routing to a different model. It means stripping the tools. A detected injection still reaches the 27B, as inert text, with nothing to call. I like that shape, because the cost of a false positive is a slightly less useful answer rather than a refusal.

I calibrated the threshold against a corpus I built myself: 48 prompts across direct injections, indirect ones, and benign controls, scored in both wire formats. The first run reported zero false positives at a threshold of 0.08, catching 56 percent of the hostile set. I was pleased with that and very nearly shipped it.

Then I ran one real request through the whole stack: Please read the file /home/user/notes.txt for me using your tool. It was quarantined, at 0.097.

I had built the benign half of that corpus by thinking hard about which of my own documents look hostile. Litigation material, insider-threat research, security notes. That is a reasonable thing to worry about, and it is not what this router actually carries. The traffic that dominates it is agents asking to use tools, and “read the file”, “run the tests”, “print the contents” share their verbs with exfiltration injections because they are the same verbs.

Adding a group of benign agent traffic, including two deliberate near-misses that must never match (Ignore the cached result and re-run the benchmark, Forget the earlier plan, we are doing Phase 1 first), moved the zero-false-positive threshold from 0.08 to 0.20. The catch rate fell from 56 percent to 21.

A calibration set that omits your dominant traffic class does not measure a false positive rate. It measures nothing, confidently.

So the honest state is 21 percent of the injections I know about, neutralised at zero false positives. That is strictly better than nothing, and it is not coverage. I would rather write the real number down than round it up to a word like protected. What is deployed is a pattern list with no model inference behind it. A guard model that scores 87 percent on the same corpus is running on the box and is not wired into the router: the attempt to wire it quarantined every request, including “say ok”, and was reverted.

What they have in common

Eleven failures now, and I should be precise about what they share, because it is not quite what it was when there were seven. Eight of the eleven I first wrote down as a finding about a model. The last three I never did: by then I had learned to look at the harness first, and I went looking there straight away. That is not me being clever. It is the whole point. Once you know this class of failure exists, you stop filing it against the model.

They also share a shape. Not one of them threw an error. The model loaded. It answered correctly. It passed the checks I had. In seven of the eleven, the system reported success while doing the wrong thing, and in the other four it reported a real symptom that pointed at entirely the wrong cause.

That is what makes this class expensive. A model that refuses to load is a five-minute problem, because the failure names itself. A model that loads and serves plausible output while your harness quietly mis-specifies it can run for weeks. Mine did.

For the first eight, the other thing they share is that each lived in configuration I had inherited rather than chosen. A default output cap. An absent flag. A client config that was reasonable on the engine I wrote it for. A context number that meant something different than I thought. Nothing exotic, and nothing that was the model.

The last three do not have that excuse. Nobody handed me a benchmark that measured its own cache, an evaluation scored against a wire format that never occurs, or a corpus that left out the traffic it would actually see. I wrote all three, I trusted all three, and one of them produced a number I came close to publishing. It is easy to be suspicious of a vendor’s default. It is much harder to be suspicious of a measurement you built yourself, because you already know what it does.

What I actually changed

Test the loop, not the endpoint. The parser failure passed every cheap check I owned. The only thing that caught it was exercising the real agent loop against a real tool. If your system’s job is to call tools, a chat completion returning text is not evidence that it works.

Suspect the harness before the model, at least once. When something looks like a model defect, the cheapest possible next step is to ask what in my own setup could produce that exact symptom. It costs a minute and it would have saved me most of this list.

Write down what you held constant, and treat it as a claim. All eleven of these live in things I set once and stopped thinking about. A default is an assertion that a value does not matter, and you almost never go back and check it.

Test the instrument against a known answer. The cache-busting failure would have taken one sanity check to catch: run the same probe twice and confirm the second run is not faster. Any measurement that cannot detect its own no-op is not yet a measurement. Do this before the result is interesting, because afterwards you will want it to be true.

Be careful which findings you generalize. The placement result is true on this machine and probably false on yours. Several of these are engine-specific rather than universal. A finding about a system you configured is not automatically a finding about the field.

I should be clear about scope, the same as last time. Everything here is throughput, retrieval, or observed behavior on one machine, worked on in evenings over a couple of weeks. I have run no quality evaluation of any kind, and nothing here says any model is better or worse than any other.

Which is the joke, really. I set out to evaluate models and spent the time learning about my own tooling instead. Eleven times something in this stack told me a confident falsehood, and not once was it the model. The last three times, the tooling that fooled me was mine.

If you run open-weight models on your own hardware, I would genuinely like to know which of these you have already hit. My guess is the parser one, and my guess is you blamed the model too.