Tag: vLLM

I wanted one place to decide which model answers and how hard it thinks. That surfaced a bug one event from the end of every stream. I fixed the wrong thing first, the schema proved the router right, and the real fix is now upstream in vLLM.

A laptop, a single GH200, and a semantic router between them, built to answer trivial requests locally and send the hard ones to the big card. Along the way, eleven things in that stack told me a confident falsehood. Not one of them was the model, and three of them were my own measurements.

I spent a night carefully A/B testing a model on a GH200. Then I upgraded the inference engine for an unrelated reason, re-ran the identical config, and got 43 percent more decode throughput and nearly four times the context budget. When open-weight architectures change this fast, the engine is the other half of the model, and it fails quietly.