METR Measured an 18% Speedup and Then Told You Not to Trust the Number

On February 24, 2026 METR published an update titled We are Changing our Developer Productivity Experiment Design. The title is dull. The content is not. They reported a speedup and then spent the rest of the post explaining why their own number is probably wrong. This is the lab whose 2025 paper — cited thousands of times — found that AI tools caused a 20% slowdown for experienced open-source developers. Now they estimate −18%. A research group being careful when the result cuts against them is expected. Being careful when it cuts their way is worth reading.

A confidence interval that crosses zero is not a headline, it is a limit

1. What It Says, In Its Own Numbers

The baseline first: METR's earlier paper found that using AI tools caused tasks to take 20% longer among experienced open-source developers, on data from February to June 2025. To see how that effect moved over time, they started a new experiment in August 2025 with 10 developers from the original study plus 47 newly recruited from a more diverse set of open-source projects — 57 people at $50 an hour. The method carried over: developers pre-specified the tasks they intended to work on, each task was randomised into an AI-allowed or AI-disallowed condition, and time on task was recorded. The results: for the subset of original developers, an estimated speedup of −18% with a confidence interval from −38% to +9%; among newly recruited developers, −4% with an interval from −15% to +9%. Both intervals cross zero.

# 1) The two estimates METR published, with their intervals.
#    Nothing here is rounded or reinterpreted.
rows = [
    # cohort,          point, ci_low, ci_high
    ("original 2025",   +19.0,  +2.0,  +39.0),   # tasks took LONGER
    ("returning devs",  -18.0, -38.0,  +9.0),    # speedup
    ("newly recruited",  -4.0, -15.0,  +9.0),    # speedup
]

for cohort, pt, lo, hi in rows:
    crosses = lo < 0 < hi
    print(f"{cohort:16} {pt:+.1f}%  [{lo:+.1f}, {hi:+.1f}]"
          f"{'  <- spans zero' if crosses else ''}")

2. Why the Number Is Unreliable: Three Reasons, From the Source

The first is participation. Through 2025 more open-source developers adopted agentic tools like Claude Code and Codex, and the direct consequence is that more of them declined to enrol, because enrolling means working without AI for half their tasks. The bias that introduces has a known direction: it pushes the speedup estimate down. The second is pay. They cut the rate from $150 an hour to $50, which is a selection effect of its own. The third is the most technical and the easiest to skip: their measurement of time spent per task is unreliable for the fraction of developers who run several AI agents concurrently. The first reason is bad news, the second is a fixable mistake, and the third points at something deeper than this study.

# 2) An interval that spans zero does not mean "no effect".
#    It means this design cannot tell you the sign. Those are different
#    claims, and only one of them is supported by the data.
import math

def half_width(sd, n, z=1.96):
    return z * sd / math.sqrt(n)

# Rough check: what sample size would pin an effect of this size down?
sd_ratio = 1.0          # spread of per-task speed ratios, generous
for effect in (0.05, 0.10, 0.18):
    n = (1.96 * sd_ratio / effect) ** 2
    print(f"to detect {effect:.0%} at 95% confidence: ~{math.ceil(n)} paired tasks")

3. Concurrency Quietly Ruins Your Internal Benchmarks

If you have ever measured how much faster tasks go with AI inside your own company, that third reason should stop you. Picture one developer driving three agents on three tasks at once. Every task's wall-clock duration now includes the time it spent waiting behind the other two. What you measured is not the cost of a task, it is the latency of a queue. This was a minor problem in 2025, when most people ran one window and one conversation. In 2026, running several agents is ordinary. So every internal number built on wall-clock time systematically overstates cost and understates speedup, and the size of the error depends on how busy the person was. No statistical correction fixes that. You have to change what you measure: active time instead of wall clock, with concurrency recorded as an explicit field.

When participation is optional, the sample is the finding

# 3) METR's own list of why the follow-up is unreliable.
#    Read this as the honest part of the paper, not the appendix.
reasons = [
    "developers decline to enrol because they do not want to work without AI",
    "that refusal biases the speedup estimate downward",
    "pay was cut from $150/hr to $50/hr, adding a selection effect",
    "time-on-task is unreliable for developers running several agents at once",
]

# Point 4 is the one that quietly kills most internal benchmarks too:
# if one person drives three agents in parallel, wall-clock time is not
# the cost of a task, it is the cost of a queue.

4. If You Still Want a Number, Make the Design Defensible

METR's response was not to give up, it was to change the design. Four things transfer to an internal team. First, the unit is a task, not a developer and not a sprint, because variance between people swallows everything else: two engineers on the same ticket can differ by a factor of five. Second, randomise: in the same task pool, half allowed AI and half not, or you are measuring task difficulty rather than tool impact. Third, write down the task list and the exclusion criteria before you run it. The difference between pre-registering and explaining afterwards is whether anyone should believe you, and it costs an hour. Fourth, make the primary metric instrumented active minutes, and add review time plus revert rate within 14 days as secondaries. That last one gets skipped constantly, because the cost of AI-written code usually lands two weeks later in review and rework rather than at write time, which is exactly the window a one-week measurement misses.

# 4) If you still want an internal number, make the design defensible
#    before you run it. Pre-registration is cheap; re-litigating a
#    contaminated number for a quarter is not.
experiment:
  unit: task                    # not "developer", not "sprint"
  assignment: randomised        # AI allowed vs disallowed, per task
  pre_registered:
    tasks:
      - "drizzle: add cursor pagination to orders list"
      - "fix: flaky integration test in billing worker"
  excluded_up_front:
    - "tasks where the dev already knows the fix"
  primary_metric: active_minutes   # instrumented, not self-reported
  secondary:
    - review_minutes
    - revert_rate_within_14d
  stopping_rule: "80 paired tasks, then report whatever it says"

5. So What Is Actually True in 2026

METR's own conclusion: based on conversations with participants, it is likely that developers are more sped up by AI tools now, in early 2026, than their early-2025 estimates suggested — but because of the selection effects, their data is only very weak evidence for the size of that increase. Look at the structure of that sentence. A directional judgement first, an explicit statement of evidence strength second. That is how an honest research group writes, and it is the filter to apply to every productivity report you read this year. The most honest answer available in October 2026 to how much faster AI makes developers is: positive direction, unknown magnitude, and a magnitude that depends heavily on task type and on how well the developer understands the tool. Anyone quoting a precise percentage without an interval is selling something.

// 5) Instrumented timing beats a survey. Log the spans you already have.
//    active minutes, not calendar minutes - concurrency has to be explicit.
type TaskSpan = {
  taskId: string;
  startedAt: number;
  endedAt: number;
  agentsInFlight: number;   // >1 means wall clock overstates the cost
};

function activeMinutes(spans: TaskSpan[]): number {
  return spans.reduce((total, s) => {
    const wall = (s.endedAt - s.startedAt) / 60_000;
    return total + (s.agentsInFlight > 1 ? wall / s.agentsInFlight : wall);
  }, 0);
}

// Then publish it with the interval, not alone:
// "review time fell 12% (95% CI 4% to 19%, n=87 paired tasks)".