Recursive Self-Improvement

Sources#

• HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution
• Jeff Dean: The 1% Rule for Building in AI
• Knowledge-Centric Self-Improvement
• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown
• Recursive Self Improvement for Coding Agents
• Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents
• When AI builds itself

Summary#

Recursive self-improvement (RSI) is the point at which an AI system can fully autonomously design and develop its own successor — closing the loop so that each model is improved by the previous model rather than by humans. The Anthropic Institute essay When AI builds itself (Marina Favaro & Jack Clark, June 2026) is this wiki's primary source. Its argument has two halves: (1) a present-tense empirical claim that AI is already accelerating the development of AI (AI Accelerating AI Development — e.g. Anthropic engineers ship ~8× more code per quarter than in 2021–2025), and (2) an extrapolation that the trend "points to an AI system capable of fully autonomously designing and developing its own successor." Anthropic's stated position: "We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for."

This page is the hub for the RSI cluster — the trajectory, the futures, and the governance response. The measured evidence lives in AI Accelerating AI Development; the capability-gating eval in AI R&D Autonomy Evaluation (AECI); the deployment brake in Responsible Scaling Policy Evaluations; and the coordination problem in Frontier Pause Verification.

Closing the loop#

The essay frames RSI as the endpoint of a steadily-tightening development loop, illustrated as person → computer → chatbot → agent → workers (each stage delegates more of the work to AI):

• 2021–2023 — Building the first Claude. Humans write code and docs on laptops; AI is absent from the loop.
• 2023–2025 — Chatbots. People paste model-generated snippets into editors.
• 2025–2026 — Coding agents. Agents write and edit whole files on their own (Claude Code launches Feb 2025).
• Today — Autonomous agents. Agents run their own code and delegate hours of work to other agents (the loop primitive running unattended).
• 20XX? — Closing the loop. "Agents could become capable enough to build and train models themselves. If this happens, future versions of Claude could be continuously improved by Claude itself." This last step is RSI.

"What if we're wrong?" — why direction-setting may not save us#

The natural objection: the work still in human hands — choosing which problems to work on (Research Taste as the Human Bottleneck) — is what matters most, so AI remains a capable assistant, not an autonomous driver of progress. The essay offers two rebuttals:

• Perspiration is becoming automated. AI advances rarely come from "eureka" moments; paradigm shifts (the Transformer, mixture-of-experts) "arrive years apart." In between, "most progress is incremental: we scale something up, see what breaks, fix it, and try again" — exactly the workflow Claude now excels at. Edison's "1% inspiration, 99% perspiration" is invoked: "we see perspiration becoming increasingly automated." Large-scale research progress "is mostly a function of tools and resources" — how fast and how many experiments you can run — which is the bitter lesson pushed to its limit.
• A conservative reading still compounds. Even if Claude never gets research taste, if humans spend most of their time on the single-digit fraction of work that is direction-setting while Claude handles the rest, each human steers far more work than before. "AI already makes Anthropic move much faster than it did before."
• The less-conservative reading. The early evidence of improving research judgment (51%→64% on next-step decisions; see AI Accelerating AI Development) suggests taste "might be just another AI capability that AI systems fail at for a time, then get good at" — the same pattern seen with explaining why a joke is funny, theory of mind, and linguistic riddles (Jagged Intelligence (Ghosts, Not Animals)).

Three possible futures#

The essay lays out three scenarios for "what happens next," contingent on whether the trend continues and what we choose to do:

1. The trend stalls (S-curve), but today's capabilities diffuse widely. Exponentials bend; the judgment separating a competent researcher from a great one may not come from scaling compute/data, requiring a new architecture past the Transformer — or the binding constraint may be the supply chain (energy, chip fab, grid, interconnect) rather than intelligence. Even frozen at today's capability, the world changes: Project Glasswing already shifted the cyber bottleneck from finding to patching (LLM-Driven Vulnerability Research), and a 100-person company can increasingly do the work of a 1,000-person one (AI-Native Startup Lifecycle). Anthropic thinks this is unlikely — "we have not yet seen that curve bend."
2. Compounding efficiency gains; humans still set direction. AI development becomes substantially automated but humans judge results. 100-person companies do the work of 10,000–100,000; revolutionizes knowledge work and government — but could power authoritarian surveillance or individualized influence ops at superhuman scale. The essay says the evidence suggests this is the likely path — bounded by Amdahl's law (below).
3. Full RSI — AI builds its successors. Pace becomes determined entirely by compute (and algorithmic-efficiency discoveries). Humans move "most of our effort towards oversight, validation, and verification of an expanding 'virtual lab' run by AI systems," with skills transferring to the rest of science. How the alignment problem resolves here is what Anthropic is "least certain about": models may be aligned and wise enough to find novel solutions (or to halt), or "the rare occurrences of misalignment present in today's models could compound as the models build their successors, growing more frequent but less understood until we lose control."

What the term is starting to get used for (and shouldn't)#

By mid-2026 "recursive self-improvement" has begun appearing as a product framing for narrow scaffold optimization, and the two senses need separating. Cline's July 2026 post Recursive Self Improvement for Coding Agents (case-study) opens by linking this essay and the Wikipedia RSI article, and reports that one prompt drove 17 hours of autonomous agent work that patched Cline's own harness — retry policy, loop detector, process handling — lifting Terminal-Bench 2.1 from 77.5% to 88.8% with Kimi K3.

That is a real result and it is not this page's subject. The model's weights were untouched; the artifact was a pull request against a TypeScript repo; the direction came from a human-written brief with a pinned end state; the campaign ran once rather than compounding; and a human reviewed the PR before merge. Nothing about it demonstrates a system designing its own successor — the definitional core above. What it does belong to is AI Accelerating AI Development, the essay's present-tense empirical half: AI compressing AI-adjacent engineering work (four engineers × two weeks in January → 17 unattended hours in July, per Cline's own before/after). The full weighing, including the overfitting caveat that the vendor has been hill-climbing this benchmark for six months, is in Agent-Authored Harness Optimization.

The distinction is worth keeping sharp because the vocabulary is the vendor's claim, not a definition, and because there is a testable boundary underneath it: does the improvement transfer? A scaffold patch that lifts held-out tasks the optimizer never saw is a different object from one that lifts the suite it was optimizing against. No transfer measurement accompanies the Cline campaign.

A controlled evaluation now runs that boundary test, and it lands on the right side of the line. HarnessBank (Luo et al., arXiv 2607.13683, empirical) evolves harnesses for frozen backbones across seven benchmarks with sealed test splits, and reports two results that matter here:

• The gains are real but the artifact is not portable. Six of seven benchmarks credit the evolved harness on held-out tasks (+9.2 to +15.4pp), yet cold-starting the loop on a different model family produces a different harness, and transplanting one model's harness onto another is near-zero off its matched failure pathology and -15.7 when the same lever is turned the wrong way. "A credited harness is a correction fitted to the model, not a universally good setting." Harness self-evolution produces model-specific corrections, which is definitionally not general capability.
• The loop converges rather than compounding. Under its significance gate the search terminates at its 10-round floor; only the ungated variants run forever, and they do so because they hallucinate progress in 62–76% of post-convergence rounds. There is no compounding here to extrapolate from — a human re-issues the brief for each new model and domain.

So the strongest-evidence source on harness self-evolution is also the strongest argument that harness self-evolution is not this page's subject. Full treatment in Agent-Authored Harness Optimization.

Compounding and RSI are orthogonal#

A third self-improvement axis clarifies the vocabulary further, by scoring better on the properties the RSI literature cares about while being further from the definition. Knowledge-Centric Self-Improvement (Wang et al., Caltech, arXiv 2607.19592, empirical) makes the agent explicitly disposable — fresh context every attempt, no private memory, no specialization, no modified prompts — and lets only an external curated knowledge base persist. Yet:

• It transfers. A knowledge bundle frozen at generation 10 lifts zero-shot solve rates on held-out tasks in every donor-recipient cell, in both cross-family directions — where a transplanted harness is near-zero off its matched pathology.
• It accumulates. Ten generations run under one protocol with no human re-issuing a brief, against harness evolution converging at a 10-round floor.

Nothing about the model improves, and the paper does not claim otherwise. Which is the point: a system can accumulate a portable, compounding asset with zero movement toward designing its own successor. "It compounds" is therefore not evidence of proximity to RSI, and the two properties should be argued separately. (The compounding claim has its own caveats — solved tasks are retired from the pool each generation, so part of the generation-over-generation curve is easy tasks leaving, and no run goes past ten generations.)

The verification leg: a closed loop still needs a grader it did not write#

The three futures above all assume, silently, that each generation can tell whether its successor is better. Guo et al. (Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents, CAS, arXiv 2607.24300, empirical) test that assumption directly in the non-gradient self-improvement setting this page's vocabulary debate lives in, and it fails: an agent that edits its policy and its own tests together keeps self-scores at 0.70–1.00 while 15 of 35 model-game policies end below the game's random reference, and the failure is stratified but never absent across the capability ladder — weaker agents overwrite behavior they had discovered, stronger ones "mismeasure the shifted deployment distribution." Constraints that stay inside the self-authored instrument do not close the gap, and an information limit (α + β ≥ 1 - TV(P+, P-)) says no endogenous-only gate can, once regressing and non-regressing candidates look alike from inside.

The paper's conclusion is a structural claim about self-improvement in general, not about Atari: "reliable self-improvement need not abandon self-verification, but it requires at least one deployment-acceptance signal outside the agent's control." Read against the definition at the top of this page, that is a constraint on what "fully autonomously designing its own successor" can mean — a genuinely closed loop would have to author its own acceptance boundary, which is the one component this result says cannot be endogenous without the improvement signal losing deployment meaning. Whether the exogenous anchor must stay human, or can be an executable procedure the system is merely forbidden to read (SEAL's audit is the latter), is the live design question. Developed on Optimizer–Evaluator Decoupling; the harness-loop instance is on Agent-Authored Harness Optimization.

A second lab's version of the same prediction — and the missing cheap validator#

Anthropic's essay is the corpus's primary source, but the trajectory is not Anthropic's alone. Jeff Dean (Google Chief Scientist, YC Startup School 2026, practitioner-opinion) was asked for a 2027 prediction and gave essentially this page's future 2, arriving from the systems side rather than the safety side: "a lot more automation of ML systems themselves… getting ML systems to improve their capabilities by running lots of experiments, breaking things down into sub-problems, running those sub-problems in a tight automatic experimentation loop, putting the results together." He generalizes the precondition — "anything where you can have a measurable objective" — and names the objective function: "optimize your discoveries per unit of compute input." Note the convergence is on the mechanism, not the risk: Dean's version has no misalignment branch and no governance response, and he never uses the term "recursive self-improvement" about it.

His contribution the Anthropic framing lacks is the rate-limiting step, and it is not the proposer. Today's model-development loop already looks like his description (small-scale experiments → promising ones scaled → results integrated into a recipe); what makes automating it interesting is loop latency, and latency is usually set by evaluation. His worked example is a decade-old quantum-chemistry result from Google colleagues: density functional theory takes a night of compute to characterize one molecule, so they trained a neural approximation on simulator input/output pairs and got something ~300,000× faster and "nearly as accurate." "Now you have 10 million things to screen. You could do that while you go to lunch rather than it being a six-month endeavor."

That cuts against the verification leg above in a specific way worth flagging. Guo et al. establish that the acceptance signal cannot be endogenous; a learned surrogate of an exogenous simulator is an interesting middle case — it is trained from the external oracle rather than authored by the improving agent, but "nearly as accurate" is precisely the regime where a screening loop learns to live in the surrogate's error. Nothing in the corpus measures what happens when an automated experimentation loop optimizes for 100,000 rounds against a 300,000×-cheaper approximation of its own validator. Dean does not raise the question; Optimizer–Evaluator Decoupling is where it belongs.

Amdahl's law for organizations#

A recurring brake across futures 2–3: speeding up one part of a process just shifts the bottleneck elsewhere; overall pace is capped by the parts that haven't sped up (Amdahl's law). Anthropic has already hit its signature: as more code flows through the org, human code review became the new bottleneck — the org-level instance of Verification as the New Bottleneck. The same friction appears beyond engineering: an explosion of ideas/initiatives/tools "far more than we have the capacity to pursue." Spotting and clearing these bottlenecks "may become the most important skill for any organization." This is also why "the felt pace of this future will still be set by the bottlenecks" — RSI can't run clinical trials faster than biology, hold elections sooner than constitutions allow, or turn a stranger into an old friend in a weekend.

An external practitioner reaches the same brake by a different route. Noam Brown (OpenAI, practitioner-opinion) argues an overnight intelligence explosion is unlikely precisely because peak capability requires large-scale test-time compute — runs that take weeks or months — so time itself becomes the binding constraint and the realistic shape is a "gradual takeoff," not an instant one. It is the Amdahl's-law point made about inference duration rather than org throughput; the full argument sits in Intelligence Explosion Dynamics.

What should we do? (the governance response)#

Anthropic argues it would "likely be a good thing" to have the option to slow or pause frontier development so societal structures and alignment research can keep up — but a unilateral pause merely changes who leads, and a real one requires multilateral, verifiable coordination. Building the systems that make a credible pause possible is the subject of Frontier Pause Verification and the Anthropic Institute's agenda. "The window to investigate the questions together is here, and people outside AI companies should be involved."

A second lab's governance answer: allocate, don't gate#

Zuckerberg's August 2026 manifesto (prediction) reaches the same trap and takes the opposite exit. He states the competitive dilemma more bluntly than Anthropic does — "any lab that doesn't let their AI system direct a substantial amount of compute capacity towards recursive self-improvement will inherently fall behind" — and illustrates the runaway with a self-improving system that finds "100x or more intelligence out of each gigawatt," at which point a fraction of world compute commands more effective intelligence "than everyone else combined."

His remedy is not a pause option but a compute-allocation ratio: labs and clouds should collectively build enough compute to spend competitively on RSI "while still committing the significant majority towards people's individual goals." Three ways it differs from the paragraph above:

• The safety property is a fraction, not a capability gate. Nothing is withheld; the claim is that RSI stays safe as long as the majority of world intelligence remains pointed at human-chosen goals. No threshold is named and no measurement is proposed — and from outside a lab, the fraction is not observable.
• It resolves the trap with capex. Anthropic's answer to "everyone must race" is verifiable coordination; this one is a larger denominator.
• It accepts unaligned goal-pursuit as the cost. "Any AI engaging in recursive self-improvement is by definition directing and advancing its own goals… not inherently harmful by itself, even if its goals are not fully aligned with many people, as long as we maintain a balance of power that favors people overall." That trades the alignment property for an unmeasured quantitative one.

Where the two agree is worth recording, because it is the stronger claim: both hold that a lab running powerful self-improving models privately is the most dangerous configuration, not the safest. Zuckerberg's version — "regardless of how much a lab rationalizes this activity in terms of responsibility and safety, this is the path of developing a singular superintelligence that cannot be checked by other systems" — is aimed at closed labs, but it is the same reason Anthropic gives for wanting the pause option to be multilateral.

Connections#

• AI Accelerating AI Development — the measured, present-tense evidence half of the essay; the data behind "the loop is tightening"
• AI R&D Autonomy Evaluation (AECI) — the capability-side gate: AECI and the substitution threshold are how Anthropic measures "can the model build the next model?"
• Responsible Scaling Policy Evaluations — the deployment brake; the RSP AI-R&D threat model is RSI risk made operational
• Research Taste as the Human Bottleneck — the last human comparative advantage; whether it holds determines which of the three futures obtains
• Task Time-Horizon Scaling — the external trendline (METR doubling every ~4 months) that makes the extrapolation quantitative
• Frontier Pause Verification — the governance response: building the verification regime a credible slowdown would require
• The Bitter Lesson — "perspiration is automatable" is the bitter lesson applied to research itself; RSI is its furthest extrapolation
• Agentic Loops Overtake Bespoke Systems — RSI's clearest existing-domain proxy: a simple loop matched a bespoke trained system as the model improved
• Harness Shrinkage as Models Improve — the same human-role-narrowing dynamic; humans stop writing code and shift to review
• Verification as the New Bottleneck — Amdahl's law instantiated: review becomes the binding constraint as generation accelerates
• Agentic Misalignment (AM) — the failure mode that could compound through self-improvement: misalignment growing "more frequent but less understood"
• Jagged Intelligence (Ghosts, Not Animals) — the "taste is just another capability AI masters" argument rests on the joke/theory-of-mind precedent
• LLM-Driven Vulnerability Research — Glasswing is the essay's proof that even frozen capability reshapes the world
• Autonomous Scientific Discovery — June 2026 wet-lab evidence that "perspiration is becoming automated" reaches discovery itself (the futures-2/3 case): autonomous drug design, novel hypotheses, week-long genomics
• AI-Native Startup Lifecycle — the diffusion scenario: each employee atop a pyramid of agents; 100-person firms doing 1,000-person work
• AGI-to-ASI Pathways — DeepMind's "From AGI to ASI" report makes RSI its pathway 3; the theory-first sibling treatment to Anthropic's empirical essay
• Intelligence Explosion Dynamics — the growth-curve question (exponential vs. hyperbolic/singularity vs. S-curve) and the four RSI mechanisms (genetic, cultural, cooperative, data), from the DeepMind report
• Multi-Agent Collective Intelligence — cooperative/sociogenic RSI: specialization in agent collectives freeing resources for further specialization
• Large-Scale Test-Time Compute — Brown's test-time-compute pacing argument: peak capability needs long runs, so time bounds the takeoff and an overnight explosion is unlikely (developed in Intelligence Explosion Dynamics)
• Agent-Authored Harness Optimization — the term borrowed for narrow scaffold hill-climbing; kept adjacent deliberately, so the vendor framing doesn't get read back into the definition above
• Optimizer–Evaluator Decoupling — the verification leg of the definition: a self-improving system's acceptance signal is the one component that cannot be endogenous without the improvement signal losing deployment meaning
• Knowledge-Centric Self-Improvement — the axis that separates compounding from RSI: a disposable-agent protocol whose external knowledge artifact transfers and accumulates while nothing about the model changes
• Jeff Dean — a second lab's 2027 prediction converging on future 2 from the systems side, plus the rate-limiting step Anthropic's framing omits: loop latency is set by the validator, and a learned surrogate (300,000× faster DFT) is how you cut it
• Researcher Uplift from Code Output — a near-term marker on this trajectory: METR's Kwa back-solves ~2.5× serial researcher uplift from Anthropic's 8×-code figure and estimates Anthropic's own 2× overall R&D threshold trips at ~3.5× researcher uplift, "which could happen in the next year or so"

Open Questions#

• Is "research taste" a true ceiling (future 1) or just the next capability to fall (futures 2–3)? The essay frames this as the single load-bearing uncertainty.
• The RSI extrapolation rests on trends staying exponential rather than S-curving — but the essay concedes it cannot rule out an architectural ceiling or a compute/energy supply-chain constraint. Which binds first? Partially answered (synthesis against DeepMind): RSI Growth Curves: Which Friction Binds First? — the three futures map one-to-one onto DeepMind's three growth shapes; the first friction to bind is the already-binding one (Amdahl's-law verification/oversight = DeepMind's embodied bottleneck), and the abstraction barrier supplies the mechanism Anthropic lacks for whether taste is a real ceiling (Future 1). Retagged #oq/now → #oq/source 2026-08-10: which friction actually binds is now an empirical question about the next capability generation, not a synthesis gap.
• If misalignment compounds through self-improvement (future 3), is AECI-gated RSP review fast enough to catch it before control is lost?

Sources#

• When AI builds itself — Anthropic Institute, When AI builds itself: Our progress toward recursive self-improvement, and its implications (Marina Favaro & Jack Clark, June 2026)
• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown — Noam Brown (No Priors, 2026-06-26), practitioner-opinion: overnight explosion unlikely because test-time-compute dependence makes time the binding constraint ("gradual takeoff")
• HarnessBank: Semantic Gene-Bank Search with Gated Verification for Agent-Harness Self-Evolution — Luo et al. (arXiv 2607.13683, 2026-07-15, empirical): §4.6 cross-model dissociation and §4.7 termination behavior — the empirical case that harness self-evolution yields model-fitted corrections that converge, not general capability that compounds
• Knowledge-Centric Self-Improvement — Wang et al. (Caltech, arXiv 2607.19592, 2026-07-21, empirical): §3 disposable agents with an external knowledge base as the only persistent object, §4.4 held-out cross-family transfer — the case that compounding and RSI are separable properties
• Jeff Dean: The 1% Rule for Building in AI — Jeff Dean with Diana Hu, YC Startup School 2026 (2026-07-30, practitioner-opinion): §"AI Systems That Improve Themselves" and §"AI That Builds Better AI" — the 2027 automated-experimentation prediction, "discoveries per unit of compute input," and the 300,000×-faster learned DFT surrogate as the way to cut loop latency. COI: Google's Chief Scientist; the quantum-chemistry result is colleagues' work recalled without citation
• Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents — Guo et al. (Chinese Academy of Sciences, arXiv 2607.24300, 2026-07-27, empirical): the verifier-deployment gap, the information limit on endogenous evidence, and the conclusion that reliable self-improvement "requires at least one deployment-acceptance signal outside the agent's control" — the constraint this places on a fully closed loop. Parse warning: its Table 7 is collapsed and not quotable; full treatment and parse notes on Optimizer–Evaluator Decoupling