ScribeOS Press
Failure as a Product
Elena Mar
Systems
Failure as a Product
Incident culture for teams that ship models
Elena Mar
Models fail differently than CRUD. Mar writes about evals that lie, silent quality regressions, and how to run an incident when the bug is a distribution, not a stack trace.
12 pages365 words$14
Opening
The bug that does not throw
A model can be wrong at 14% and still return 200. That is the original sin of shipping learned systems. Your monitors will be green while your users quietly stop trusting you.
Traditional SRE looks for errors, saturation, and latency. You still need those. You also need a quality signal that is allowed to page someone. If the only quality process is a quarterly eval notebook, you have a research hobby, not a production system.