Empirical Study • >10M LLM Calls • 400K CPU Hours • arXiv:2611.23068

The Resolution Hypothesis

Agents already propose real improvements.
Evolution stalls because evaluators are flipping coins.

Yuanhao Zeng1,2,3,* Shaocheng Lan1,2,* Ao Lu1,2,* Kuang Yan2 Heng Lin2 Xinang Xu2 Yexin Li3 Ruo Shan2 Kan Ren1,†
1 ShanghaiTech University 2 ByteDance, China 3 BIGAI
* Equal contribution · † Corresponding: zengyh2025@shanghaitech.edu.cn, renkan@shanghaitech.edu.cn
arXiv:2611.23068 PDF Blog Read This First ↓
Highlight 01 · Proposal Works
+11.4 pp

Good skills already exist.

Sample 100 candidate skills once: genuine cross-task winners are already waiting in the pool (Quality@100 = 61.8%).

Highlight 02 · Identification Fails
49.7%

Evaluators just flip coins.

Execution gates (49.7%) and LLM judges (45.4%–49.0%) pick no better than chance—turning training gains into test drops.

Scroll down: invest in an RSI agent. Beat a coin flip. (90s) ↓
50%HEADS
50%TAILS
Bet you can't beat me.

Can you beat a coin?

🪙 RSI Ventures
Round
1/8
📈 Loop says
+0.0 pp
🔒 Real (held-out)
🔒??.?
⚡ Compute
24
What just happened → the paper

Your game was our experiment.

Every pitch was a real SKILL.md from our Terminal-Bench runs. Self-improving (RSI) agents pick upgrades exactly the way you just did.

Start reading ↓
01Preface

Before You Read

  • Why

    We audit the measurement instrument, not the ambition of self-improving agents. Method rankings flip across studies, so we ask a foundational question: when an agent edits its prompts, skills, or harness, can standard evaluation reliably tell a true improvement from a lucky run?

  • Core conclusion

    The bottleneck is identifying true improvements under noise—not proposing them.

  • Intuition
    Var(Ŷ) = σ2 + τ2

    Frozen-weight edits act through noisy multi-turn rollouts. Rollout noise (σ²) and run-to-run drift (τ²) set an evaluation resolution floor; when gains sit below it, greedy acceptance harvests luck instead of better skills.

  • Not claiming

    We are not rejecting agent evolution or questioning prior researchers' good faith. Proposers work, and even shortcuts confirm that evolution optimizes whatever signal clears the noise floor. Progress calls for four shifts:

    • Low-variance process checks: programmatic behavioral diagnostics show 13×–57× less noise than binary verdicts.
    • Component-wise evaluation: benchmark proposers (Quality@K vs. null) separately from selectors.
    • Resolution-aware reporting: publish σ̂ and τ̂ alongside point scores.
    • Higher-resolution regimes: more validation rollouts or deterministic program-environment feedback.
Next: if noise blurs improvements, why do papers keep reporting contradictory wins—and huge jumps on some tasks? 02 Introducing Our Work ↓
02Setup

Introducing Our Work

Why does evolution stall on some benchmarks while claiming +10–40 pp elsewhere? Methods trade wins across papers—stalling on Terminal-Bench while jumping elsewhere. The Resolution Hypothesis: evolution accumulates gains when they exceed evaluation resolution, and drifts on noise when they fall below it.

🔍 ZoomFigure 1: noise landscape of agent evolution and the evolution loop with predictions P1-P3
Fig. 1. Per-step gains sit inside the single-run noise band, yielding predictions P1–P3.
P1 · Proposals exist

Proposers find real gains

Candidate pools hold artifacts that improve held-out quality beyond luck.

P2 · Selection fails

Selectors sit near chance

Execution scores and trajectory LLM judges fail to rank candidates reliably.

P3 · Errors compound

Noise accumulates over rounds

Greedy acceptance inflates in-loop scores while runs diverge.

150
configs (7 methods, 6 models)
1,030
evolution runs (8–16× eval)
>2.1M
trajectories (>10M LLM calls)
3.4 pp
median lit. margin (171 results)
17.6→0.8
median pp after eval repair
49.7%
1-rollout selection accuracy
So what did 150 configurations and 1,030 runs actually reveal? Here is the 60-second answer. 03 Conclusion ↓
03Findings

Conclusion

  1. 01

    Single-run rankings rest on luck.

    Across 171 results in 9 papers, the 3.4 pp median margin sits below single-run SD on >60% of pairs; best-of-4 accuracy on Terminal-Bench is 33% (random 25%).

  2. 02

    Reported 10–40 pp gains reflect evaluation shortcuts.

    Auditing 12 benchmarks × 3 models, a one-sentence rule recovers a median 90% of gains >3 pp; repairing the evaluation drops gains from 17.6 to 0.8 pp.

  3. 03

    Proposers find real gains; selectors sit at chance.

    Quality@100 reaches 61.8% (+3.4 pp over null; shared skills span >15 pp), while 1-rollout (49.7%), 2-rollout (48.6%), and LLM judges (45.4%–49.0%) sit near 50%.

  4. 04

    Greedy selection errors accumulate across rounds.

    In-loop validation rises +6.9 to +9.0 pp while 8× validation re-evaluation shifts +0.35 to −3.21 pp, and τ̂2 grows to 4.2–24.5 pp².

Skeptical about shortcut claims, selection budgets, or whether this is just “common sense”? Read our direct responses next. 04 Q&A ↓
04Q&A

Questions We Expect

Q01Are you claiming agent evolution doesn't work, or that prior papers were flawed?

Neither—proposers genuinely work, and prior results reflect an unmeasured noise floor rather than bad faith. Tested proposers produce real held-out improvements (Quality@100 reaches 61.8%), and single-run reporting became standard before rollout variance (Var(Ŷ) = σ² + τ²) was quantified.

Q02High variance makes it hard to iterate or identify improvements—isn't that just common sense?

In principle, yes—yet in practice the community has operated on the opposite premise, and the severity is surprising.

  • Standard reporting ignores it: across 171 results in 9 agent-evolution papers, 0 report variance (σ̂ or τ̂) and only 6 average >1 evolution run, treating 3.4 pp margins as reliable rankings.
  • Intuitive selectors still fail: 1-rollout (49.7%), 2-rollout (48.6%), and LLM trajectory judges (45.4%–49.0%, r = 0.054) all sit near chance even when proposers yield +3.4 pp Quality@100 headroom.
  • Why this surprises researchers: even knowing noise exists in theory, the community rarely anticipated how severely it bites in practice—that true skill gains already sit below current selector resolution.
Q03Our method achieved +20–40 pp on a published benchmark—why call that an evaluation shortcut?

Finding a +20–40 pp lever is real optimization, but it reflects a benchmark-wide convention rather than fine-grained task skills. Under frozen weights, one global rule—like matching a strict parser or benchmark variant—lifts most tasks at once (one sentence recovers a median 90% of gains >3 pp). After evaluation repair, median gains drop from 17.6 to 0.8 pp.

Q04What about Meta-Harness, AHE, ReasoningBank, GEPA, or SkillOpt—are all 7 methods equally affected?

The bottleneck is shared across both text-artifact and harness families whenever step gains fall below resolution. Across 4 text methods (GEPA, SkillOpt-Lite, Trace2Skill, ReasoningBank) and 3 harness methods (HarnessLens, Meta-Harness, AHE) on 6 models (150 configs), no single proposer is singled out—all hit sparse terminal feedback when ranking +1–3 pp refinements.

Q05Why test 1–2 rollouts or LLM judges? Wouldn't averaging more rollouts fix selection?

Standard loops use 1–2 rollouts to stay tractable; scaling to 16× validation helps, at 16× inner-loop cost. Under standard budgets, execution (49.7% / 48.6%) and LLM judges (45.4%–49.0%) sit near a coin flip. Raising validation to 16× lifts mean test score 61.4% → 64.4% and best run 64.8% → 72.7%, motivating cheaper low-variance diagnostics.

Q06Isn't Quality@K rising with K just the “winner's curse” over noisy scores?
  • Equal-Quality Null: subtracts out the exact max-of-K curve expected if all 100 skills shared identical true quality.
  • Disjoint held-out tasks: across 10 disjoint test tasks (16 rollouts each), Quality@100 reaches 61.8% (+3.4 pp above null).
  • Single shared-skill spread: across 160 test rollouts per skill (2h budget), single skills span >15 pp (57.5% best vs. 49.2% mean and 41.9% weakest).
Q07Deterministic program evolution (e.g., KernelAgent, AlphaEvolve, ARC-AGI) compounds steadily—does that contradict you?

It supports the Resolution Hypothesis by structurally reducing evaluation noise. Evolving task-specific code enables cheap parallel re-evaluation of deterministic programs and richer runtime diagnostics, allowing incremental gains to clear resolution—unlike frozen-weight text/harness edits scored by stochastic multi-turn rollouts.

Q08Will stronger frontier models naturally solve evaluation resolution?

Across 6 models up to Kimi-K3, GLM-5.2, GPT-5.6, and DeepSeek-V4-Flash, noise shrinks mainly where benchmarks saturate. On ALFWorld, stronger models show lower noise only near the ceiling where little headroom remains. On unsaturated Terminal-Bench, median evaluation noise stays around 3.5–4.0 pp across model tiers.

Q09What should benchmark authors, method builders, and reviewers do differently now?
  • Report σ̂ and τ̂: publish evaluation noise and run-to-run variance alongside point scores.
  • Decouple proposer from selector: score proposers via Quality@K vs. null, and selectors via selected-artifact curves on fixed pools.
  • Use low-variance behavioral checks: programmatic intermediate diagnostics cut evaluation noise by 13×–57×.
Ready to inspect the four-step evidence chain—and test our numbers interactively? 05 Experiments ↓
05Evidence

Four Steps Through the Evidence

Follow the argument step by step—from method-ranking instability to why evolution loops stall—and expand any explorer to test our empirical numbers yourself.

Step I · The puzzle
Question 1 · Can a single evolution run tell two methods apart?

Single-run rankings sit below resolution

Across four audited benchmarks, >60% of same-benchmark method pairs differ by less than one single-run SD—picking the best of four methods on Terminal-Bench is right only 33% of the time (random: 25%).

🔍 ZoomFigure 2: method pairs versus one single-run SD and required runs
Fig. 2. Method-pair gaps vs. 1 SD, and runs required to resolve them.
Interactive Explorer · Run budget calculator
N ≥ 2 · [(z1−α/2 + z1−β) / Δ]2 · (τ2 + σ2)80% power, α = 0.05
Target margin Δ3.4 pp
Q1 1.2median 3.4Q3 6.6
Evaluation noise σ3.87 pp
Run-to-run SD τ1.36 pp
Runs per method N1 run
Single-run SD
–
Runs needed (80%)
–
Signal-to-noise
–
Achieved power
–
Wait: if typical method gaps sit inside noise, how do some papers report +10 to +40 pp gains? Step II · Audit +40 pp wins ↓
Step II · The illusion
Question 2 · Do +10–40 pp gains refute low resolution, or confirm it?

Large reported gains come from evaluation shortcuts

Across 12 benchmarks × 3 models, a one-sentence rule reproduces a median 90% of gains >3 pp; repairing the evaluation drops median gains from 17.6 to 0.8 pp.

🔍 ZoomFigure 3: one-rule reproduction and post-repair collapse of reported gains
Fig. 3. Reported gain vs. one-sentence rule vs. same artifact after eval repair.
Interactive Explorer · Gain audit (12 benchmarks × 3 models)
Reported Baseline + 1 rule After eval repair
Once shortcuts are removed, do genuine task skills even exist—or can LLMs simply not propose them? Step III · Proposal vs. Selection ↓
Step III · The split
Question 3 · If evolution stalls without shortcuts, does proposal fail or does selection fail?

Proposals hold real gains; selection sits at chance

Quality@100 climbs to 61.8% (+3.4 pp above the equal-quality null; shared skills span >15 pp), yet execution selectors (49.7% / 48.6%) and LLM judges (45.4%–49.0%) stay flat near 50%.

🔍 ZoomFigure 4: Quality@K versus equal-quality null, and selection versus oracle
Fig. 4. Quality@K vs. null (left); selectors vs. oracle and random (right).
Interactive Explorer · Proposal & Selection pools
Timeout regime
Candidate budget KK = 100
Quality@K
–
Equal-quality null
–
Headroom above null
–
Quality@K (95% band) Equal-quality null Genuine gap
If every single selection step is near a 50/50 coin flip, why do evolution curves look like they climb inside the loop? Step IV · Why runs drift ↓
Step IV · The drift
Question 4 · What happens when a chance selector runs round after round?

Noise accumulates into optimism drift and variance diffusion

Greedy acceptance harvests upward noise: in-loop scores rise +6.9 to +9.0 pp while 8× re-evaluation shifts +0.35 to −3.21 pp, and τ̂2 grows to 4.2–24.5 pp².

🔍 ZoomFigure 5: in-loop versus 8x re-evaluated gain
Fig. 5. In-loop vs. 8× re-evaluated gain.
🔍 ZoomFigure 6: tau squared growth over evolution rounds
Fig. 6. τ̂2 growth over rounds.
Interactive Explorer · Optimism drift & variance diffusion
In-loop gain
–
8× re-evaluated
–
Optimism gap
–
Final τ̂2
–
(a) In-loop vs. re-evaluated gain (pp)
(b) Run-to-run variance τ̂2 (pp²)
Curious what a genuinely helpful—yet invisible to selectors—skill actually looks like inside? 06 Inspect concrete skills ↓
06Case study

Inspecting Concrete Skills

Good evolved skills exist—exact-identifier and tokenizer-fidelity rules that transfer across tasks.

Inspect artifacts · seed-027 / seed-099 / seed-058
07Citation

BibTeX

@article{zeng2026resolution,
  title={The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements},
  author={Zeng, Yuanhao and Lan, Shaocheng and Lu, Ao and Yan, Kuang and Lin, Heng and Xu, Xinang and Li, Yexin and Shan, Ruo and Ren, Kan},
  journal={arXiv preprint arXiv:2611.23068},
  url={https://arxiv.org/abs/2611.23068},
  year={2026}
}