Morgan Stanley AI @ Work Assistant STATE Teardown: Evals Are Not Traces

Morgan Stanley runs one of the most disciplined eval programs in enterprise AI. Across four public sources, there is no documented way to reconstruct what the assistant told one advisor about one client yesterday.

What Is the Morgan Stanley AI @ Work Assistant?

Morgan Stanley runs the deployment everyone cites when they want to prove a bank can ship LLMs: a suite of OpenAI-powered internal tools. The flagship AI @ Morgan Stanley Assistant is a GPT-4 RAG chatbot over roughly 100,000 firm research documents; per the ZenML LLMOps writeup, over 98% of advisor teams actively use it. CNBC covered the 2024 expansion of AskResearchGPT into the institutional securities division, reporting that close to half of the firm's 80,000 employees use generative AI tools built with OpenAI.

The third product, Debrief, is the meeting tool. ZenML describes the machinery: it processes Zoom recordings with Whisper, OpenAI's speech-recognition model, and summarizes them with GPT-4. Morgan Stanley's own press release describes the workflow: with client consent, it generates notes in client meetings, saves a note into Salesforce, and creates an email for the advisor "to edit and send at their discretion." The whole program is organized around evaluation: pre-deployment evals, human-graded summarization tests, and a daily regression suite of sample questions.

The STATE Score

Scored from public evidence only, Morgan Stanley's AI @ Work program lands at 4/10 on the STATE rubric. A documented human gate on every client-bound output earns the board's only 2/2; structure, tolerance, and per-interaction traceability sit at or near zero because no source documents them.

What Morgan Stanley Gets Right

The human gate is real, documented, and quotable. The Debrief press release says the tool "creates an email for an Advisor to edit and send at their discretion," and ZenML confirms that "advisors review and adjust AI-generated outputs before finalizing them." Nothing reaches a client unreviewed. The Assistant and AskResearchGPT are retrieval tools whose outputs trigger no automated action.

That earns the only 2/2 on the board: every model output bound for a client has a deterministic stop in front of it, a human with authority to edit or discard. The eval program deserves credit too; human-graded evals and a daily regression suite are real engineering, not compliance theater. The problem is what those evals are being asked to stand in for.

Gap 1: Can Anyone Replay Yesterday's Answer?

When an LLM system misbehaves in production, the first question is: what exactly did it receive, what did it produce, and when? If you can't answer that after the fact, you're flying blind.

Everything documented about observability here is aggregate. ZenML describes a daily regression suite that "identifies potential weaknesses and ensures the system continues to deliver compliant outputs." That is quality control, and good quality control. It is not an incident record. No source describes capturing what a specific production interaction actually was: which chunks were retrieved from the 100,000-document corpus for that query, what prompt the model saw, what it returned to that advisor at that moment.

The distinction matters because a test suite is gameable in a way a trace is not. Sample questions are chosen, graded, and tuned until they pass. Production questions are not. Subtle questions, hedged answers, questions where a word like "could" does load-bearing work: when one of those goes wrong for a real advisor, would it even be logged? Nothing in the sources says yes.

So the failure plays out like this. An advisor reports that yesterday's answer on a client question was wrong. There is no trace to replay. The investigation reconstructs the interaction from the advisor's memory, then re-asks a non-deterministic model and gets a different answer. The post-mortem ends at "the model did something weird," in a firm where those answers feed regulated client conversations.

Score yourself:

Could you reconstruct exactly what happened in any specific LLM call from the past 30 days?

Gap 2: What Did the AI Contribute to That Salesforce Note?

In regulated environments, "the model did something unexpected" is not an acceptable explanation. Auditability means answering for one specific decision, not gesturing at a control environment.

The controls here are attested at the program level. ZenML notes the regression testing serves "dual purposes: maintaining quality and ensuring compliance," and the press release states employees using the tools "are bound by all applicable Morgan Stanley policies and procedures." Those are program statements. What no source describes is a per-decision record: something that links a specific AI-drafted Salesforce note or client email to the transcript span it came from, the model's raw output, and the advisor's edits.

OpenAI's zero-data-retention policy does not close this gap, because it was never about this gap. Per ZenML, it addressed concerns about proprietary data being used to train public models. That is step zero, the entry ticket for putting client data near a third-party model at all. A regulator examining one disputed client interaction is not asking about training data. They are asking what the system contributed to this record and what the human changed, and the human gate, for all its strengths, leaves no diff behind. The firm can show its controls existed. It cannot show what the AI actually wrote.

Score yourself:

If a regulator asked what data drove one specific decision last month, could you answer in under 30 minutes?

Gap 3: What Happens When Debrief Dies Mid-Pipeline?

A system that only works in the forward-motion state is a demo. Production systems fail at step 3 of 5, and what happens next is the architecture.

Debrief is a genuine multi-step workflow: recording, Whisper transcription, GPT-4 summarization, a Salesforce note, a draft email. The press release describes the happy path in detail. No source mentions a checkpoint, a retry policy, idempotency, or resume behavior at any step. Silence in vendor-friendly sources is not proof of absence. But this series scores what the public record shows, and on resilience the record shows nothing: a 0/2 that Morgan Stanley could overturn tomorrow by publishing one paragraph about its failure path.

The consequence is concrete. A failure between transcription and the CRM write leaves a client meeting's record in an unknown state. Either the advisor discovers days later that no note ever reached Salesforce, or a blind retry runs the pipeline again and files a duplicate. The sources give no basis to say which one happens, and that is itself the finding: for a tool whose whole job is making sure client meetings leave a reliable record, the failure path is undocumented.

Score yourself:

If your workflow died at step 3 right now, does it resume from step 3, or have you never tested it?

What Good Looks Like

Two additions would move this system from 4/10 to 8/10, and neither touches the model.

First, a trace row written for every production call before the answer renders, with an idempotency key so Debrief's pipeline can resume instead of duplicating:

create table llm_traces (
  trace_id       uuid primary key default gen_random_uuid(),
  workflow_id    uuid not null,       -- one per interaction (or per Debrief meeting)
  step_name      text not null,       -- 'retrieve' | 'generate' | 'crm_write' | 'email_draft'
  model_version  text not null,
  prompt_ref     text not null,       -- template version + content hash
  retrieved_ids  text[],              -- chunk ids pulled from the research corpus
  raw_output     jsonb not null,      -- the model's output, pre-edit
  human_diff     jsonb,               -- what the advisor changed before sending
  created_at     timestamptz not null default now(),
  unique (workflow_id, step_name)     -- retries resume; the CRM write can never file twice
);

That one table carries three pillars: T (any interaction replays exactly), A (raw_output plus human_diff is the decision record a regulator actually wants), and Tol (the unique key makes retries safe by construction).

Second, per-fact citations in the advisor UI. Every claim in an answer or a meeting summary should link to the source document and passage, or the transcript span, that produced it. That converts advisor verification from minutes of searching into seconds of clicking, which is the difference between a human gate that works and one that gets rubber-stamped at 4:55 on a Friday.

The Generalizable Lesson

Evals and traces answer different questions. An eval suite tells you the system is healthy in bulk; a trace tells you what happened to one user, once. Mature AI programs conflate them because both live under "observability," and the eval side demos better. But regulated industries run on the second question, and incidents are always singular. The principle to take back to your team: aggregate quality measurement is not a substitute for per-interaction reconstruction, and no amount of the first buys you the second. A mid-tier model with full traces beats a frontier model you cannot replay. State beats intelligence. The Ramp teardown found the mirror image: a system built to prove each decision rather than just run the workflow. Every teardown in this series scores from public evidence only, and this is the pattern that shows up most.

FAQ

Is Morgan Stanley's AI assistant safe for regulated data?

The documented boundary controls are strong: OpenAI's zero-data-retention policy addressed the concern of firm data training public models, per ZenML, and every client-bound output passes a human first. The Assistant and AskResearchGPT are retrieval tools whose outputs trigger no automated action, and Debrief's draft email goes through an advisor with authority to edit or discard it before anything reaches a client. What public sources do not show is interaction-level auditability, so "safe" currently rests on internal records the firm has not described.

Does Morgan Stanley's AI log LLM decisions?

Not at the per-call level, as far as any public source shows. The documented observability is aggregate: pre-deployment evals, human-graded tests, and a daily regression suite of sample questions. No source describes capturing the retrieved documents, assembled prompt, and output of a specific production interaction. That distinction decides what an incident investigation looks like. Aggregate evals answer whether the system is healthy in bulk; only a per-call trace answers what one advisor was told, once. Without one, reconstructing yesterday's answer means working from the advisor's memory and re-asking a non-deterministic model that may not repeat itself.

Do aggregate evals satisfy audit requirements for AI systems?

No. Evals demonstrate a control environment; audits ask about individual decisions. Quebec's Law 25, for instance, requires explaining a decision based exclusively on automated processing when the person asks: the personal information used, the reasons, and the principal factors and parameters that led to it. A per-decision record does what a test suite cannot: it links one specific output to the inputs that produced it, the model's raw text, and the edits a human reviewer made before anything was saved or sent. An eval suite cannot answer for interaction N. Only a per-decision record can.