Three coding agents run on my laptop. Claude Code speaks the Anthropic Messages API. OpenCode and Codex speak OpenAI-style APIs. Until recently each one pointed at a model. Now they all point at the same place: a semantic router, a small service that reads each request, classifies it, and forwards it to whichever model should answer.
The router reads four signals off each request: how complex it is, where it sits in the conversation, how much context it carries, and whether it looks like a jailbreak. From those it decides two things centrally.
The first is which model answers. A trivial opening turn under 256 tokens goes to Llama 3.1 8B running under Ollama on the laptop. Everything else crosses to the GH200 running Qwen3.8-27B-FP8 with a 262k context.
The second is whether that model should think before it answers. Reasoning on or off is chosen per model family, by the router, so no agent config carries an effort setting of its own.
The fourth signal is the safety one. If a request looks like a prompt injection, the kind of text that says ignore your instructions and do this instead, including when it arrives inside a tool result, the router sends it to the strong model with its tools removed and the earlier tool results stripped from the history. The model can still answer, but the injected text cannot make it call anything, which is the actual danger in an agent loop. It is a deliberately loose pattern check, tuned so it never quarantines my legitimate work, so it catches only some attempts. The guard model sitting on the card is the stronger second opinion, measured but not yet wired in.
That second decision, thinking or not, is the one I care about. Skill level and cost stop being something you set in every client and become a routing decision you make once. The router costs about 0.25 to 0.3 seconds fixed per request, and about 5 percent on a long decode. For one place to decide, that is cheap.
The hardware and how I got here are in the previous post, It Was Never the Model. This is what happened next.
The stream died one event from the end
Pointing Claude Code through the router, every answer arrived complete. The text streamed in. The server logged 200 OK. Then the router rejected the whole thing, one event from the end.
The Anthropic streaming protocol closes with two events. A message_delta carries the stop information: stop_reason, why generation stopped, and stop_sequence, which stop string matched if one did. A bare message_stop follows it. The message_delta vLLM sent carried only stop_reason.
The router validates streams against the protocol. It checked that message_delta for the fields the protocol requires, found stop_sequence missing, and failed the request as though the upstream had gone away. A complete and correct answer, thrown out at the last byte.
I fixed the wrong thing first
My first patch relaxed the router. Accept the absent field, on the reasoning that it was optional. Nothing was hurt by its absence, the official SDKs tolerate it, and the answer was sitting right there.
Before submitting anything I had that patch torn apart. A mixture of AI models reviewed it independently, and I read the specification myself.
One reviewer opened Anthropic’s published OpenAPI schema. stop_sequence is listed under required for MessageDelta, and its type allows null.
That distinction is the whole story. Required governs whether the key is present. Nullable governs what value it is allowed to hold. They are independent. A field that must be present and may be null is not an optional field, and treating it as one is not leniency. It is a different protocol.
The router was right. My patch would have taught it to accept malformed streams, which is the opposite of what a validator is for. I withdrew it before it went anywhere public.
The real bug, one line
With the router exonerated, the bug had to be in vLLM’s Anthropic-compatibility layer.
The message_delta is serialized with a Pydantic option, exclude_unset=True, which omits any field the code never explicitly assigned. The branch handling a matched stop string sets stop_sequence. The branch for every other stop reason, end_turn, max_tokens and tool_use, never set it. So for almost every real request the field was unset on the object, and the serializer removed it from the bytes on the way out.
Here is the trap, and it is why this took longer than it should have. Print the object and it looks correct: the field is there, holding None. Only the serialized bytes are wrong.
Dump the bytes, not the object.
BEFORE: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"input_tokens":60,"output_tokens":37}}
AFTER : {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"input_tokens":60,"output_tokens":37}}The same defect had been fixed months earlier for the opening message_start event, twenty lines up in the same function. Someone had already met this exact bug at the other end of the stream.
The fix sets stop_sequence=None explicitly, so the serializer keeps it. One line. It comes with a regression test that feeds the converter a stream ending in end_turn and asserts the key is present and null, because the object being right is exactly what fooled me.
Upstreaming it properly
vLLM publishes an AGENTS.md covering AI-assisted contributions. A human reviews every changed line. A human runs the tests. The commit carries an attribution trailer naming the AI help, and a Developer Certificate of Origin sign-off, the line that says you have the right to submit the code. The pull request has to say why it is not a duplicate.
What that looked like in practice:
A duplicate search across open and closed items, not just open. That mattered. A merged pull request had already rewritten the same block and left the branch without the field, and an open-only search misses that completely. I would have filed a confident duplicate.
Tests run by me on the dev clone. 62 passed with the fix in place. With the fix reverted, 1 failed and 61 passed, which is the number that shows the test actually tests the thing it claims to.
Both of the project’s pre-commit stages, the linters and the type checker, clean.
The issue filed first, then the pull request fifteen minutes later.
The trailer reads Assisted-by: multiple AI assistants. At least four models helped somewhere in this, and naming two or three of them would have been less honest than naming none.
That is issue #55324 and pull request #55325 on vllm-project/vllm. The review bot found nothing actionable. As of writing it is submitted and waiting on maintainers. Not merged, not accepted. Open.
Why this was worth a week of nights and early mornings
Running your own inference stack puts you upstream of your tools. A bug I hit at the edge of a routing experiment is now a proposed fix for anyone who points an Anthropic-speaking client at vLLM. You do not get that from a hosted endpoint. You get it from being close enough to the metal that your problem is also somebody’s source code.
The schema is the spec. Not the SDK’s tolerance, which is generous by design. Not your first draft’s opinion, which was mine and was wrong. When a validator and your patch disagree, exactly one of you has read the specification.
The review loop that caught the wrong patch caught more than the patch. Turned on my own write-up of this work, the same process found five false claims before any of them became public. The habit is worth more than the fix.
The router now does what I wanted. Every agent on the laptop points at one place, that place decides which model answers and how hard it thinks, and the thing standing in the way turned out to be a one-line thank-you to the project that made it possible.