01 / Blog

Notes from production.

Failure taxonomies, defensive patterns, and the architecture decisions that keep LLM systems from falling over at 2am.

FEATURED

Meta-Prompting in Production: One File the Agent Can Touch, One Verifier It Can't

One AI writing prompts for another sounds like leverage until you watch it fail: an 80-node workflow that repeated every mistake, re-paid every failed run, and trusted every generated instruction. Tuning the meta-prompt never fixed it. A three-layer contract did: a verifier the generator can't touch, domain facts it can't ignore, generation left free in the middle. Two production failures in its first 24 hours stress-tested exactly that.

The Meta LayerJuly 15, 202610 min read

Filter by pillar

Production Failure Taxonomy STATE Framework Applied Defensive Architecture The Meta Layer Regulated AI & Law 25 Teardowns

TeardownSTATE Framework AppliedJuly 21, 202610 min read

Morgan Stanley AI @ Work Assistant STATE Teardown: Evals Are Not Traces

Morgan Stanley runs one of the most disciplined eval programs in enterprise AI: pre-deployment evals, human-graded tests, a daily regression suite. The STATE score is still 4/10, because nothing in the public record can reconstruct what the assistant told one advisor about one client yesterday. This teardown dissects the gap between aggregate evals and per-interaction traces, and shows the single table that closes it.

TeardownSTATE Framework AppliedJuly 6, 20269 min read

Ramp Financial Automation Agent STATE Teardown: Graduated Autonomy Proves the Decision, Not the Workflow

Ramp runs one of the most disciplined AI agent rollouts in production finance — graduated autonomy, human escalation, 99% policy accuracy. STATE score: 6/10. Across three public sources, there is no evidence the approval workflow survives a crash mid-flight.

Production Failure TaxonomyMay 8, 20266 min read

The AI Didn’t Hallucinate. It Told the Truth. That Was the Problem.

A technically solid LLM email pipeline — 10,000 emails, 36 classification buckets, six N8N workflows — reproduced institutional confusion at scale and called it procedures. This is what Knowledge State Failure looks like, and why no model upgrade fixes it.