Tag: Inference

At my day job, I was asked to be more efficient with AI spend. The honest answer was a semantic router in front of every model, so I pointed my own coding agent at one. It broke. Five bugs later, the fixes are upstream, which means everybody gets them.

A normal model is twelve editors who each read your draft once. A looped model is one editor who reads it twelve times. I spent this week working out what that buys you, what it costs, and where it stops working.

I wanted one place to decide which model answers and how hard it thinks. That surfaced a bug one event from the end of every stream. I fixed the wrong thing first, the schema proved the router right, and the real fix is now upstream in vLLM.

A laptop, a single GH200, and a semantic router between them, built to answer trivial requests locally and send the hard ones to the big card. Along the way, eleven things in that stack told me a confident falsehood. Not one of them was the model, and three of them were my own measurements.

I spent a night carefully A/B testing a model on a GH200. Then I upgraded the inference engine for an unrelated reason, re-ran the identical config, and got 43 percent more decode throughput and nearly four times the context budget. When open-weight architectures change this fast, the engine is the other half of the model, and it fails quietly.