Products · Deployment & application
Developer productivity uplift (METR RCTs)
1What this measures
Measured speed change for experienced developers using AI tools versus not, from METR's randomised trials. Negative means slower. The band applies to the newest estimate (Feb 2026, newly recruited developers).
Why it matters. The single best-controlled test of whether frontier capability turns into product-level uplift. The paper's claim is that benchmark gains do not translate quickly; this is where that shows.
- Proxy types
- product, deployment
- Unit
- share
- Cadence
- annual
- Valve
- invention to product
2How we track this
- series
metr_blog.rct_*.dev_speed_change.pt - source METR research posts · default tier 6 · METR
- Normal band
- ≤ 20.0%
- Fast band
- ≥ 40.0%
- Falsifying
- —
Normal = uplift of at most about 20%, setting-dependent (the range of controlled studies to date); fast = broad uplift above 40%, the scale the productivity-boom claims need.
Applied to metr_blog.rct_2026.dev_speed_change.pt.
3Tracker interpretation
Two trials, two years, and the best estimate is still near zero. Capability is not the constraint; integration and verification are.
4Evidence
5Status and reasoning
METR's Feb 2026 redesign estimates a -4% speedup for newly recruited developers (CI -15% to +9%; METR calls it very weak evidence); the 2025 RCT found developers 19% slower. Both are far below the 20% ceiling of the normal band. Initial seed.
6Timeline notes
7Counterevidence
What cuts against this reading
Small samples; the 2026 redesign has selection effects METR itself calls 'very weak evidence'; experienced open-source maintainers on their own repos are the hardest case for uplift.
8Update history
- 2026-09-10unmeasured to consistent with normalconf — → 70 · claude-initial-seed
METR's Feb 2026 redesign estimates a -4% speedup for newly recruited developers (CI -15% to +9%; METR calls it very weak evidence); the 2025 RCT found developers 19% slower. Both are far below the 20% ceiling of the normal band. Initial seed.
9Confidence
70 / 95 — good evidence, some ambiguity
Confidence is independent of status: 90–95 multiple strong independent sources; 70–89 good evidence, some ambiguity; 50–69 mixed or hard to operationalise; below 50 limited or vague.