---
title: "One Control Point"
description: "At my day job, I was asked to be more efficient with AI spend. The honest answer was a semantic router in front of every model, so I pointed my own coding agent at one. It broke. Five bugs later, the fixes are upstream, which means everybody gets them."
author: "Colin McNamara"
tags: ["Inference","vLLM","Open Source","Architecture","Agents"]
image: "https://colinmcnamara.com/images/blog/semantic-router-upstream/social-card.png"
date: 2026-09-28
url: https://colinmcnamara.com/blog/one-control-point
---

# One Control Point

Every CIO I work with has a chart.

*Figure: A schematic line chart with no numbers on it. Two lines, AI usage and token spend, climb up and to the right together over time, spend tracking usage. Both continue as dashed lines into a shaded region on the right labeled next quarter, marked with a question mark.*

It shows AI usage climbing, and token spend climbing with it, on infrastructure they are accountable for. The line goes up and to the right, and the people drawing it are the people being asked what it costs next quarter.

At my day job, where I am a Field CTO at AHEAD, that question arrived as a simple request: be more efficient. It was a reasonable request, and it is where this story starts.

My own question was narrower. Can the tools pick the right model and reasoning level for each request, automatically? Not a human choosing from a dropdown, and not a config file somebody updates quarterly. A decision made per request, by something that has read the request.

## One control point

That something is a semantic router. It sits in front of your models and reads each prompt before deciding where it goes. One place, three jobs.

**Cost.** Send each request to the cheapest model that can actually handle it. Many requests do not need your most expensive model.

**Availability.** Move models in and out behind a stable name. Swap a backend without downtime, because callers address the router, not the model.

**Security.** Screen prompts before a model sees them. Jailbreak and PII signals at the front door rather than after the fact.

In most enterprises I work with, those are three projects owned by three teams. Putting them at one control point is the interesting part, not any one of them alone.

*Figure: Claude Code and other apps and agents send requests through a gateway into one Semantic Router, the single control point that handles cost, availability and security. The router sends this prompt to the small model, one of three models (small, large and a hosted provider) that sit behind one stable name.*

The router is not the only routing decision a request meets. There are three, and it helps to keep them apart.

*Figure: A request moves down through four layers: Client, Gateway, Inference platform and Model replicas. Semantic Router sits beside the Gateway, connected by an ext_proc gRPC stream, and is marked as the part PR #4266 changed. Three callouts show a separate decision at each layer: the Gateway picks which service, Semantic Router picks which model, and the inference platform picks which replica.*

[The project's own documentation](https://vllm-sr.ai/docs/installation/k8s/inference-platforms) draws the same three-way split, and it is worth borrowing even if you never use this software. Its advice on the first layer is the practical kind: [use the gateway your platform already operates](https://vllm-sr.ai/docs/installation/k8s/gateways). You are not supposed to adopt a new data plane to get smarter routing.

## I routed my own work through it, and it broke

I did not evaluate this by reading about it. I put my own work behind it.

My coding agent is Claude Code. I pointed it at vLLM Semantic Router, and the Router at vLLM running on a GH200 machine in my lab. Personal hardware, because my work laptop was not powerful enough for the models I wanted behind it. That detail matters later.

Then I used it for real work, on the path that gets less testing. Claude Code speaks the Anthropic Messages API, `POST /v1/messages`, not OpenAI's Chat Completions. Most routing examples and tests use Chat Completions. Two of the bugs below, and the open one at the end of this post, are on it.

*Figure: Claude Code sends POST /v1/messages through a gateway (Envoy or agentgateway) to Semantic Router, then on to vLLM on an arm64 GH200. Five green numbered pins show where each fixed bug sat: two stop_sequence bugs on vLLM and the Router, buffered response diagnostics and arm64 builds on the Router, and my own fix on the gateway-to-Router link, where full duplex dropped the routing decision. Two of the five, the vLLM stop_sequence fix and the full duplex fix, are my own merged code. An amber ring on vLLM marks the one still open, null stop fields in non-streaming /v1/messages.*

| Bug | Issue and fix | Who fixed it |
|---|---|---|
| `stop_sequence` missing from `message_delta` in streaming `/v1/messages` | vLLM [#55324](https://github.com/vllm-project/vllm/issues/55324) → [#55325](https://github.com/vllm-project/vllm/pull/55325) | My fix, merged by Robert Shaw, a vLLM Lead Maintainer |
| Router rejected responses that lacked `stop_sequence` | [#3500](https://github.com/vllm-project/semantic-router/issues/3500) → [#3944](https://github.com/vllm-project/semantic-router/pull/3944) | Xunzhuo, using the lenient-with-diagnostic design I recommended |
| Buffered response diagnostics never reported | [#4118](https://github.com/vllm-project/semantic-router/issues/4118) → [#4125](https://github.com/vllm-project/semantic-router/pull/4125) | Xunzhuo |
| arm64 builds failed on `libgomp.so.1` | [#4132](https://github.com/vllm-project/semantic-router/issues/4132) → #4125 | Xunzhuo. Caught on my arm64 Mac; the GH200 is arm64 too |
| Full duplex dropped the routing decision | [#3892](https://github.com/vllm-project/semantic-router/issues/3892) → [#4266](https://github.com/vllm-project/semantic-router/pull/4266) | My fix, merged 2026-09-28 |

I wrote up the first of those in [The Router Was Right](https://colinmcnamara.com/blog/the-router-was-right). Short version: a field was missing from a stream, I wrote the wrong fix first, and the schema settled it.

The three in the middle were fixed by the maintainers and shipped in the v0.4 "Hermes" release on 2026-09-27. To be precise about my part: three bugs I reported were fixed in v0.4, and the maintainers wrote that code. My first merged Semantic Router code is #4266, which landed after the release.

## The fix I am proudest of

The plain-English version of #4266: the routing decision was made, and then it was thrown away.

### What broke

In full duplex mode (`FULL_DUPLEX_STREAMED` in Envoy, `FullDuplexStreamed` in agentgateway), the gateway streams the request body to the Router without waiting for replies. That is the point of the mode, and it is efficient. Here is what main did with it.

*Figure: Sequence diagram of the failing path. The gateway streams request headers to the Router, which replies to the headers immediately before reading the prompt. The gateway then streams the body. The Router picks the model at the end of the body, but its header changes arrive too late to take effect and are dropped in red, so the request goes to the default backend with the client credential.*

The Router replied to the request headers immediately, before it had read a byte of the prompt. It then read the body, picked the model at the end of it, and set the routing headers: `x-selected-model`, the provider path, the provider credential.

But a body reply cannot change headers in this mode. The header stage was already over. Envoy said so in its own log line:

```
NOT clearing route cache. No header mutation in the response
```

The result on main was not a subtle degradation. Requests went to the default backend carrying the client's own credential, even when the body was small enough to arrive in a single write. A request with trailers got a 500 from Envoy with `Spurious response message`. Through agentgateway, the backend rejected the body as invalid JSON and returned a 400.

### What fixed it

The design that fixed it was not mine. YAMAMOTO Takashi, who reported #3892, proposed it in the issue: hold the header reply until the body has been routed. Then send it, carrying the routing headers and the route cache clear, then the body reply, then a trailers reply if the request has trailers.

*Figure: Sequence diagram of the fixed path. The Router holds its header reply, shown as an amber bar, while the body streams in. It picks the model, then sends the held header reply carrying the routing headers and the route cache clear, then the body reply, then trailers. The request reaches the selected backend on the provider path with the provider credential, shown in green.*

What makes that legitimate rather than a hack is that [Envoy's specification explicitly permits it](https://github.com/envoyproxy/envoy/blob/ff3fe7f0bb9e4ac6a283ea38bf9ee3f375530d56/api/envoy/extensions/filters/http/ext_proc/v3/processing_mode.proto#L92-L94): a server may wait for the body before answering the headers. The mode was never the problem. The Router answered too early to act on what it learned from the body.

With the fix, the selected backend, the provider path and the provider credential all arrive correctly, on both gateways. The diagrams show a Chat Completions request from the project's E2E tests; the fix covers any request streamed in full duplex, `/v1/messages` included.

My favorite detail is in main's own source: a comment noted that the header changes "have no effect," and the code then threw them away. It was a known limitation, written down in the code. The fix moves those changes onto the held reply, where they do have effect.

### Choosing a mode

Here is the whole trade-off:

*Figure: A two-by-two grid rates each way a gateway can hand the request body to the Router: efficient on the horizontal axis, correct on the vertical. BUFFERED is correct but not efficient. Headers only, STREAMED (sometimes wrong) and full duplex before #4266 are efficient but not correct. An arrow labeled #4266 lifts full duplex into the green goal cell, the only mode that is both efficient and correct.*

Two operational notes came out of review and are now in the documentation:

- **On agentgateway,** routing on Router-set headers requires the ExtProc policy in `traffic.phase: PreRouting`, because the default `PostRouting` picks the route before the Router speaks.
- **Keep `failure_mode_allow: false` for full duplex.** If the stream fails while the header reply is held, failing open would forward the client's original headers, which is the situation you were trying to prevent.

## The people who fixed it

This is the part I want to be specific about, because "the community fixed it" is not a credit, it is a shrug.

**Xunzhuo Liu** (@Xunzhuo, AMD) is a Semantic Router maintainer and, in practice, its technical lead: 463 merged PRs, and a co-lead of Data Plane & Networking. Xunzhuo accepted #3500 within minutes and built the design I had recommended, then fixed #4118 and #4132 for v0.4. Xunzhuo also asked for the end-to-end (E2E) coverage, and reopened #3892 and marked it in progress for me, which is why I got to write the fix at all.

**Stefan Wang** (@1fanwang, LinkedIn) trimmed my PR description to the project's style, specified the E2E test exactly, and gave the first approval: "LGTM. Thanks for the quick turnaround." Stefan is a committer, one of four Data Plane & Networking co-leads and a lead of Agentic & Context.

**Abhinav Mahajan** (@abhinav-m22, Barclays), a committer, asked whether a held reply survives a stream error. It was the best question on the PR, and the answer found a real gap. I fixed it, posted the evidence, and Abhinav approved.

**Wilson Wu** (@wilsonwu, DaoCloud), a committer and a lead of Developer Experience & Ecosystem, approved.

**YAMAMOTO Takashi** (@yamt) reported #3892, diagnosed it, proposed the hold-the-header-reply design, and approved the implementation with "lgtm thank you."

**Binbin Zhang** (@Bevisy, China Telecom) updated the branch so it could merge. Binbin is a committer, a lead in three workgroups and a Kata Containers maintainer.

On the vLLM side, #55325 was my own patch, 29 lines across two files. **Robert Shaw** (@robertgshaw2-redhat, Red Hat), one of vLLM's five Lead Maintainers, merged it, and **Sylvester Kaczmarek** (@sylvesterkaczmarek) reviewed and approved it after checking CI.

*Figure: A timeline from Sep 4 to Sep 28, 2026, drawn to scale, with one bar per bug from filing to fix. My fix for #55324, PR #55325, was merged by Robert Shaw in 6.5 days. Semantic Router #3500 was accepted in 6 minutes and fixed by #3944 in 15 days. #4118 and #4132 were both fixed by #4125, in about a day and about 12 hours. My fix for #3892, PR #4266, merged 1.9 days after I claimed it, the day after v0.4.0 Hermes was released.*

There is a lesson in that timeline, and it is a lesson rather than a grievance. This team moves on reports, which is the whole reason to file them here. It is also why you should claim an issue first if you want to write the code yourself. I claimed #3892 at 13:01Z and opened #4266 at 23:55Z the same day. Claim to merge was 1.9 days.

## How to do this yourself

Six things, in the order they mattered.

**1. Prove the bug on real software before you claim it.** #4266 was demonstrated against a real Envoy 1.35.3 and a real agentgateway v1.3.1, not a mock.

**2. Write tests that fail on main and pass with the fix.** The routing tests in #4266 did. Then plant deliberately wrong fixes and confirm the tests catch them.

*Figure: Two bars drawn to the same scale compare the size of the fix in PR #4266 with the size of its tests. The fix is 187 lines of router code. The tests are 1,157 lines: 407 unit test lines, 707 E2E test lines and 43 test backend lines. That is about six lines of tests for every line of fix, and they caught 23 of 23 planted wrong fixes.*

**3. Have several AI models review the work independently, then check their findings yourself.** They disagree with each other, which is useful. They are also confidently wrong on occasion, which is why the checking is not optional.

**4. Own the reviewer's catch in public.** Prove exactly where the reviewer was right, fix it, and post the evidence in the thread. Do not quietly patch it.

**5. Build exactly what the maintainers ask for.** Stefan specified the E2E test. The E2E test matched the specification. That is most of what "easy to review" means.

**6. Credit everyone by name.**

## Full circle

I needed this capability for my own work. Stated plainly, the capability is this: optimize token spend in real time, and move models in and out transparently without taking services down. Because the fixes went upstream instead of into a private fork, it is not mine alone.

Leaders watching that chart have a lever on spend worth measuring. AHEAD's engineers, sellers and project managers can use the same tools I use. Everyone else who runs this stack gets it for free, including the people who will hit that full duplex bug next month and never know it was there.

That is the argument for working upstream rather than around it.

> A private fix helps one company, and that company maintains it forever. An upstream fix helps everyone, and somebody else maintains it with you.

*Figure: A clockwise loop of six steps. A question about rising AI spend leads to one control point that routes each prompt to the right model, then to me running my own work through it, where it broke with five bugs along the way. The bugs were fixed upstream with the vLLM and Semantic Router maintainers, so everyone can use it, and a dashed amber arrow back to the start reads next: measure the savings.*

## What is next

**Measure it.** I have not measured the savings. This is where the work laptop comes back in. The before-and-after token number, taken there, is the actual answer to the question that started this, and I have not run it yet. I would rather publish the mechanism now and the measurement when it exists than imply a number I do not have.

**Bugs in flight.** My patch for null stop fields in non-streaming `/v1/messages`, vLLM [#56376](https://github.com/vllm-project/vllm/pull/56376), is open and waiting on review. And the Router's finish-reason allowlist rejects values that vLLM itself can return. That one is next, once I have captured it on my own hardware.

**Research directions**, and I mean research rather than product. The control point is the natural place to screen for prompt and logic injection, because it is the one place every request passes through. AI honeypots are the other thread I keep pulling on. Neither is something I am shipping. Both are things I think the control point makes newly possible.

The chart still goes up. The difference is that there is now somewhere to stand.