ScribeOS Press
The Inference Floor
Amira Chen
Architecture
The Inference Floor
Serving systems that stay boring under load
Amira Chen
A working architecture for teams who put models in production and then have to live with them. Chen writes from on-call, not from a keynote: batching, caches, fallbacks, and the politics of a p99 that will not move.
Opening
Boring is a feature
The first duty of an inference platform is not cleverness. It is to be boring at 2 a.m. when a product surface is on fire and the only question that matters is whether the model is still answering.
I have sat in rooms where the architecture diagram was a cathedral and the production path was a ladder leaning against it. The diagram had vector stores, agents, evaluators, a feature store, and a square labeled "guardrails" that no one could point to in the repo. The ladder was a FastAPI process, a Redis list, and a human who knew which canary to revert.