back
Twice as Fast, Mostly Redacted
EdgeFiles

Twice as Fast, Mostly Redacted

About the Author

Sources

METR found Anthropic accelerating, and the public got the verdict without the evidence

In August, Anthropic told the public that AI was making its research significantly faster, “but not yet by a factor of 2.” The private version of the same report, which an outside evaluator got to read, was less sure. Its central estimate sat just below two, and the company thought that estimate more likely too high than too low. Then it added that “it seems plausible we do exceed this factor of 2 overall.”

The evaluator was Model Evaluation and Threat Research (METR), a Berkeley, California-based nonprofit that tests frontier AI models. Working from figures in the private report, METR calculated that one of Anthropic’s leading indicators implied an acceleration of 1.97.

It suggested that calling the result roughly two would have told readers more than calling it less than two. That gap is the story.

Not the speedup itself. Dario Amodei, Anthropic’s CEO, had already named it as his first concern in September, when he argued that AI was increasingly building the next generation of AI and called on the industry to slow down. The story is how little of the evidence anyone outside the company gets to see, and how late.

If a model helps build its successor, the time between an internal advance and its public disclosure stops being a communications delay. It becomes the window in which oversight either happens or doesn’t.

The interval between generations could shrink even as the lab’s lead over the people assessing its risks grows.

What METR was allowed to see

From August 12 through October 29, 2026, Anthropic gave METR benchmark scores for unreleased models, internal training metrics and time with researchers who could explain what the numbers meant. The access exceeded METR’s expectations, even though Anthropic could decline individual requests. METR worked from records and interviews and neither requested nor received access to try the unreleased models itself.

One of the most useful dates in the assessment is the day a model became available to most Anthropic employees. For a model the company never released, that date puts it on the same timeline as the models the public eventually got to use. It also shows what an outside observer misses by watching launches. A model can already be at work inside the lab, helping with the work that will shape its successor, while the public record still ends with the previous generation.

METR kept sole control of its methods and conclusions. Anthropic kept control of the evidence.

In its November 5 assessment, METR said it could share its highest-level conclusions, but Anthropic had redacted most of the evidence the evaluators found most informative. METR could tell readers where a redaction mattered. It could not show them what was behind it.

Sam McCandlish, an Anthropic co-founder and its responsible scaling officer, wrote on X that many of METR’s conclusions matched his own. “AI is speeding up our research, but it’s hard to say exactly how much,” he wrote in a post reported by the Associated Press. He acknowledged that the redactions made METR’s work harder to check and argued that the details were commercially sensitive and could show other labs how to go faster.

METR didn’t endorse that reasoning. It recorded Anthropic’s rationale, that the redactions targeted confidential information that could help other developers accelerate, and noted that the result left readers relying on METR’s judgment “much more than would be ideal.”

Around twice as fast, give or take a lot

Most of the METR team concluded that, as of October 2026, Anthropic was likely improving its models on well-specified tasks about twice as fast as it had from March 2024 to October 2025. The team was not unanimous. It said a rate closer to 1.3 times or three times the historical pace would not have surprised it, and the judgment rested on more than benchmarks, including evidence METR saw and the public cannot.

The benchmark part comes from Anthropic’s version of the Epoch Capabilities Index (AECI), a composite score designed by the research group Epoch AI that combines results from many benchmarks. In METR’s main analysis, AECI scores rose at about 1.9 times their earlier rate after October 2025. The confidence interval, 1.5 to 2.8 times, only covers uncertainty in each model’s score, not the choice of which models went into the fit. Change the assumption about how task difficulty is distributed and the estimate lands anywhere from 1.1 to 2.9 times, and METR said most of those alternatives fit the data about equally well.

Two things pull in opposite directions. Newer unreleased models made a lasting change in the trend look more plausible than a one-time jump. A relatively slow stretch for Anthropic before late 2025 could also make the later increase look larger than it is.

That is a lot of uncertainty around a number that carries policy weight. The evidence is strongest where a task can be specified clearly and its result graded. Solving more of those tasks is valuable in a lab, but how far the gains carry into open-ended work remains an open question. An experiment can have a clean answer and still leave the researcher with the harder decision of what to try next.

Feeling faster is not building faster

One researcher told METR that people used to skip certain experiments because each idea took a day or two to implement, and that AI assistance had led them to try far more ideas. Another named hiring as probably the biggest factor behind the speedup. In informal Slack polls, where staff voted with emoji reactions, they said on average that the most capable internal model made them about 12 times faster at their own work in October. METR treated that as very loose evidence, and it answers a different question.

Feeling twelve times more productive at your desk does not mean the company’s models are improving twelve times faster. On that question, METR’s answer was about twice the old rate.

A poll response about personal productivity says little about how fast the company can produce a more capable model. An experiment that now takes an afternoon to set up can still wait for a training run, and ten quick experiments can still yield nothing worth keeping.

METR confidently ruled out faster scaling of training compute as the explanation. It judged that factors other than AI probably played a large role in late 2025, and that by October 2026 AI likely accounted for most of the increase, roughly 80 percent by its own subjective estimate. That second judgment came with substantial uncertainty. Anthropic’s own staff had no consensus on how much of the speedup came from AI, and outsiders cannot see some of the evidence METR used to weigh the competing explanations.

For now, METR’s account is mainly about AI helping people do research, though staff described early signs of more autonomous work in some parts of model training. The more consequential possibility is that each generation takes on enough of the next development cycle to shorten it further. METR found no convincing bottleneck that would stop that compounding, but its view of the bottlenecks changed several times as Anthropic supplied more information.

The pace of research can change before the result shows up in a trained model. Part of the next cycle may already be underway when the benchmark registers the gain.

Who pulls the trigger

In September, while METR’s assessment was underway, Amodei called on the industry to slow improvements in model capabilities so safety measures could catch up. His proposal began with embedded external evaluators. The practical test of that idea is whether an evaluator can see enough, early enough, to contest a company’s decision about when more scrutiny is warranted.

Anthropic’s Responsible Scaling Policy (RSP), the voluntary framework the company wrote to decide when rising risk demands stronger safeguards, shows why the reported near-doubling cannot settle that decision on its own.

Under the policy’s acceleration criterion, Anthropic must observe or expect progress in aggregate AI capabilities at twice both the expected rate and the fastest extended rate it has seen without significant AI contributions to AI research.

That historical rate has to span at least three model generations. Automation of research or engineering must plausibly explain a substantial part of the doubling, and continuing the trend must seem likely to accelerate progress further. A separate criterion covers models that could fully substitute for Anthropic’s research scientists and engineers at competitive cost. METR’s October estimate compared progress on well-specified tasks with one stretch of Anthropic’s past. The policy asks for a broader measure and different comparisons.

Crossing the line matters because of who gets to look afterward. The RSP requires Risk Reports, public assessments of the company’s catastrophic risks, every three to six months. If one covers a model past the automated research threshold and is “significantly redacted,” the policy calls for a full external review. The Long-Term Benefit Trust, an independent body of trustees with the power to elect members of Anthropic’s board, can also request an outside review.

Now look at who pulls those triggers. Full external review applies only once Anthropic has determined that a model crossed its own threshold. Whether a report counts as significantly redacted is a call for the responsible scaling officer, the CEO, the board or the trust. The responsible scaling officer is McCandlish, the executive who just defended this round of redactions.

Every trigger sits inside Anthropic’s own governance. That makes it a commitment, not a check.

Anthropic’s August Risk Report, which assessed models as of July 15, shows how much room for disagreement exists before any trigger fires. Anthropic judged catastrophic risk from automated research and development to be low and concluded that its models met neither criterion. It said it was less confident than in earlier reports and acknowledged that its measurements lag, which makes very recent acceleration hard to see.

In a separate review from the same engagement, also published November 5, METR agreed with the low-risk judgment but found the public case for staying below the acceleration threshold overconfident. Most of the relevant researchers at both organizations leaned toward the threshold not having been crossed by July 15. METR still considered a crossing plausible. That is where the 1.97 came from. The public report described its leading indicators as below double, which was technically true and gave a reader no reason to think the estimate might be that close.

What still goes unchecked

The scope of a review is another source of leverage. The acceleration project originally included a check of Anthropic’s compute allocation, especially between capabilities and safety. Anthropic cut it midway, before METR reached any substantive conclusions, saying it wanted to keep the engagement focused on risk assessment for now and would revisit the check later.

Anthropic has published its own snapshot. Over one week in July, about 6 percent of the compute going to AI research and development went to safety work. That figure went without independent verification in this assessment. Anyone judging whether safety can keep pace with capability gains would start exactly there.

Anthropic’s separate October arrangement with an embedded METR team goes further than the acceleration assessment did. Anthropic says six METR staff now use internal tools and systems roughly comparable to those of its own risk assessment teams, with some exceptions. They can publish without a company veto, though Anthropic can still require redactions in predefined categories and decides what falls into them. According to Reuters’ account of the announcement, METR can publicly question a redaction without revealing what was withheld. The team can also make certain statements about the access it received, and other METR staff are kept separate from its work.

That is real progress. It still leaves the question that matters most. If the lab and the evaluator disagree about what the public needs to know, who else can examine the evidence?

An audit a second team could run

Here is what I would want before trusting anyone’s acceleration number, Anthropic’s or a competitor’s.

Start with the record. An evaluator needs a dated history of internal model snapshots, including models the lab never launches, and enough benchmark data to rerun the analysis under different assumptions. A common set of tests gives reviewers a point of comparison across labs. A separate set of tasks held back from training shows whether gains extend beyond familiar evaluations.

To test how much of the speedup came from AI, reviewers need to see when teams gained access to better tools and what work they handed off.

Anthropic has already prototyped a measure that follows a fixed basket of research tasks and rates how much of each AI performs. A shared version would show how work changes across labs. Where teams doing similar work got a new tool at different times, a reviewer can compare what changed after each rollout and check whether hiring or compute changed in the same weeks. Failed experiments belong in the record too. They tell a reviewer more than a retrospective tally of breakthroughs.

None of this requires publishing trade secrets. The records can sit in a secure environment where a second qualified team draws its own samples and reruns the analysis. A lab can keep the details of a training breakthrough confidential while letting reviewers report the period covered, the evidence they examined, and how far their conclusion moved when they changed the baseline. The public record can carry the disagreements and name the missing evidence that would resolve them. A policymaker could then tell a finding that survived another team’s scrutiny from one whose most important assumptions remain out of reach.

An audit that only the audited lab can check is not oversight. It is disclosure on the lab’s terms.

A threshold also needs a procedure for the days when a crossing is plausible, and the evidence is still being tested. Anthropic’s own measurement proposal floats a “fixed testing window” before a new model is used for further AI research. It should be more than a suggestion. Opening that window on preliminary evidence would give independent reviewers a chance to test whether the threshold had been crossed while the lab still had a decision to make.

Anthropic’s private report already called a crossing plausible. The public learned that in November, from someone else.

By the time the next model makes the acceleration visible, the decision the audit was meant to inform may already have been made.