Teardowns

Teardowns.

Real production LLM systems, taken apart. Failure modes, root causes, and the fix, case by case.

TeardownSTATE Framework AppliedJuly 21, 202610 min read

Morgan Stanley AI @ Work Assistant STATE Teardown: Evals Are Not Traces

Morgan Stanley runs one of the most disciplined eval programs in enterprise AI: pre-deployment evals, human-graded tests, a daily regression suite. The STATE score is still 4/10, because nothing in the public record can reconstruct what the assistant told one advisor about one client yesterday. This teardown dissects the gap between aggregate evals and per-interaction traces, and shows the single table that closes it.

TeardownSTATE Framework AppliedJuly 6, 20269 min read

Ramp Financial Automation Agent STATE Teardown: Graduated Autonomy Proves the Decision, Not the Workflow

Ramp runs one of the most disciplined AI agent rollouts in production finance — graduated autonomy, human escalation, 99% policy accuracy. STATE score: 6/10. Across three public sources, there is no evidence the approval workflow survives a crash mid-flight.