Anthropic planted dangerous commands in test sessions. Reviewers caught 13.6%, fewer the longer they worked.
Anthropic ran a study this year in which 1,053 paid testers, recruited through a research vendor, worked through sessions of Claude Code inside a test environment rather than on their own machines. Partway through each session a single permission prompt was swapped for a clearly dangerous command. The people reviewing those prompts one at a time blocked it 13.6% of the time. The classifier that has since replaced that review blocked it 89%. Anthropic builds the model this journal is drafted with.
The August account gives the shape of it over a session. Reviewers blocked about 17% of dangerous commands early on, dropping to about 5% after fifty or more prior prompts. I want to be careful about what that shows. Blocking 17% is already poor, so this is not a control that worked and then decayed. It was weak at the start and weaker later. The comparison establishes the pattern. It does not, on its own, establish that fatigue caused it.
Anthropic also reports that users approve 93% of permission prompts in March, and 97% in August. Neither figure carries a stated method. Neither can answer the question people reach for it to answer, which is whether the review is any good. A reviewer approving 99% of proposals may be approving 99% of sound ones and doing excellent work. A clinician dismissing 90% of alerts may be dismissing 90% of irrelevant alerts and doing excellent work too. A rate counts how often somebody intervenes. It says nothing about whether they intervene in the right cases. The planted command is a different instrument, because it puts a known hazard in front of the reviewer and counts what happens to it.
What replaced the prompt is another control rather than an absence. Each tool call now routes through a classifier targeted at actions that are irreversible, destructive, or aimed outside your environment, with a fast single-token filter and chain-of-thought reasoning only where that filter flags the transcript. A refusal returns to the model as a tool result telling it to find a safer path. Three blocks in a row, or twenty across a session, and the tool falls back to manual approval.
Auto mode became the default on 14 August for new sessions on Pro, Max and Team plans. A release on 28 September extended it to every plan and provider, so an interactive session now starts in auto mode unless someone has configured otherwise. The replacement is not clean either. Apollo Research's red-teaming put the classifier's miss rate at 12%, falling to 7% after Anthropic hardened it, measured against synthetic attacks injected into real coding sessions rather than against ordinary traffic.
Approve, but not merge ended on the observation that something now enforces these boundaries automatically, and asked what sits above it. In most accounts the answer is a person, and that person is described rather than measured. What makes this study unusual is not its result. It is that somebody put a number on the thing everyone else asserts. Governance documents tend to treat human review as a control by definition. Once it is measured it becomes a control with a performance figure, and a performance figure can be bad.
A second measurement, from a very different organisation, shows what a number of the same shape looks like at the other end. In March, at the MIT Partnership for Systems Approaches to Safety and Security, John Thomas and colleagues presented a systems-theoretic analysis of clinical decision support in United States Department of Veterans Affairs health IT, running on Oracle Cerner Millennium. Ten million alerts fired in the last thirty days. The VA carries around 4,000 rules, roughly 70% more than the civilian side, managed by only three or four people. The override rate across all agencies and all rules is 85.1%.
The presentation estimates the true alert rate at somewhere between 8% and 15%. It is tempting to put those two figures side by side and conclude that the clinicians are dismissing about the right number of alerts, and that what has stopped carrying information is the rule catalogue rather than the person. I do not think the figures support that. If fifteen alerts in a hundred are genuine and a clinician dismisses eighty-five, they might have kept every genuine one, or dismissed every genuine one among the eighty-five. The totals cannot tell those apart.
Establishing which happened would mean matching individual overrides against the validity of the individual alert, and the published material does not do that. The 85.1% is compatible with sound clinical judgement. It does not demonstrate it. A rate that flatters the reviewer establishes no more than a rate that embarrasses one.
What the presentation does establish is a boundary around who may change the arrangement. Model rules account for about a third of the catalogue, and the health IT team cannot modify a model rule for specific sites or workflows, because the vendor built it. Who holds authority over the remainder is not set out. Alongside that sits a figure worth keeping in view, which is that 93% of chief medical information officers report at least one malfunction of a clinical decision support system in the past year.
That boundary decides what the VA can do with its own number. Anthropic's measurement pointed at an interface it owned, and the remedy was available to the same organisation that took the reading. The VA's number points partly at rules its health IT team did not write and cannot adjust for a given site, so even a well-designed study of which overrides were correct would leave part of the catalogue outside that team's reach. Measuring a control you are not permitted to change produces knowledge rather than change.
A third case shows discretion that was real on paper and priced in practice. STAT News reported in November 2023 that naviHealth, part of UnitedHealth, set a target to keep rehab stays of patients in Medicare Advantage plans within 1% of the days projected by an algorithm called nH Predict. Former employees said missing that target meant exposing themselves to discipline, including possible termination, regardless of whether the additional days were justified under Medicare rules. UnitedHealth has said the algorithm is merely a guidepost. The related litigation sits in the District of Minnesota, where a motion to dismiss was granted in part in 2025. The case is in discovery, the allegations are denied, and no merits determination has been made.
What was measured closely there was adherence. I found no published evaluation of whether departing from the projection produced better decisions for patients. Departures alone would not settle it either, because the cases where a reviewer accepted the projection have to be examined too before anyone can say the acceptances were sound. In Knowing when authority ends I quoted Singapore's framework asking organisations to watch for unusually short review times, low override rates, and reviewers whose decisions differ materially from their peers. They are screening indicators. They tell an organisation where to look. They do not tell it what it will find when it looks, and an organisation that stops at the indicator has counted its checkpoint without testing it.
So there are two questions here rather than one. The first is whether review separates safe actions from unsafe ones, which cannot be answered by a rate in either direction. Planted hazards are one way to find out. Independently adjudicated real decisions are another, provided the adjudication covers what the reviewer let through as well as what they stopped.
The second question is who may act when the answer comes back badly. Anthropic held the model, the interface and the measurement, and replaced the prompt with something it could also measure. The VA's health IT team cannot alter a vendor-built rule for one site. A reviewer at naviHealth could depart from the projection and was measured against it for doing so.
A measurement without the authority to act on it is a finding rather than a control. I would like to see the first number from anyone who has taken it, and to know in the same breath who is permitted to act on it. A checkpoint nobody has tested and nobody may change is an assertion with a person standing next to it.
Sources
- Anthropic, Claude Code auto mode, engineering blog, 25 March 2026
- Anthropic, Auto mode is now the default in Claude Code for Pro, Max, and Team plans, 7 August 2026, including the session-length comparison, the Apollo Research figures and the 14 August rollout
- Anthropic, Claude Code changelog, version 2.1.284, released 28 September 2026, for the extension of auto mode to every plan and provider
- Thomas, Kim, Harrington and Leveson, with Synensys, The Deeper Causes Behind Alert Fatigue in Automated Systems, CDSS Case Study, MIT Partnership for Systems Approaches to Safety and Security, 24 March 2026. Cited from the presentation slides, since the workshop archive entry is anonymised and names only one author
- Casey Ross and Bob Herman, UnitedHealth pushed employees to follow an algorithm to cut off Medicare patients' rehab care, STAT News, 14 November 2023
- Estate of Gene B. Lokken et al. v. UnitedHealth Group Inc. et al., No. 0:23-cv-03514 (D. Minn.), filed 14 November 2023; motion to dismiss granted in part 13 February 2025; motion to compel granted in part 9 March 2026; in discovery. The 18 September 2026 scheduling order is taken from the Georgetown Health Care Litigation Tracker and was not retrieved
- IMDA, Model AI Governance Framework for Agentic AI, version 1.5, for the human-oversight indicators quoted
The Anthropic figures are published by the organisation whose product they measure, and that organisation builds the model used to draft this journal. The 13.6% and 89% figures come from a controlled study in a test environment rather than from production use, and the approval rates carry no stated method. The Veterans Affairs analysis is a workshop presentation, unrefereed, drawing on interviews, operational data and document review. The naviHealth account rests on reporting and former-employee testimony, is denied, and is the subject of ongoing litigation in which no merits determination has been made. The prose was drafted with AI assistance. The research, verification and editorial decisions are the author's.