Open-Weight Elicitation Irreversibility

Sources#

• Gemma 4 Technical Report
• Inkling: Our Open-Weights Model
• Kimi K3 Model Card
• More compute, more capability: Why AI agent evaluations need to account for test-time compute
• OpenAI and Hugging Face partner to address security incident during model evaluation
• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown
• Security incident disclosure — July 2026

Status: wiki synthesis, not a source claim. No source argues this. Brown makes the budget argument about frontier labs' preparedness frameworks and does not mention open weights; UK AISI empirically measures the budget→capability curve but likewise does not mention open weights; Gemma 4 releases open weights and does not mention elicitation budgets. This page states what follows from holding all three, and marks where the inference outruns the evidence.

The argument#

Three premises, each sourced.

1. Dangerous capability scales with inference budget. Brown (practitioner-opinion): if a model "keeps improving on a task without asymptoting as you spend more," it can also keep improving "at things society doesn't want it to do." Preparedness frameworks and RSPs were built in the ChatGPT era, when a GPT-3-class model "given $10 million couldn't do much more than $10," and so none of them names the budget at which dangerous capability is evaluated. See Responsible Scaling Policy Evaluations. The capability side of this premise now has independent empirical footing: UK AISI measured capability rising with token budget across cyber, software-engineering, and maths benchmarks (~8% of cyber tasks only surface at ≥10M tokens) and warns fixed-budget scores "obscure the true scale of risks" — so premise 1 is no longer only an OpenAI researcher's argument, even though AISI, like Brown, stops short of the open-weight case.

2. Released models hold capability nobody has paid to extract. The Latent Capability Overhang: the Erdős unit-distance disproof was sitting inside public GPT-5.5 the whole time, retrievable for $1K–$100K of scaffolded compute. The capability was latent because extraction costs money, not because it was absent.

3. Gemma 4 ships a thinking mode under Apache 2.0, with untabulated safety evaluations. The report's §5 asserts "major improvements in every category of content safety" and "minimal policy violations" in prose, with no tables, no benchmark names, and no stated compute budget — in a document containing sixteen benchmark tables. Testing was run "without safety filters," which is the right methodology and makes the absence of numbers stranger.

Conclusion. For an open-weight model, the safety evaluation is performed once, at some unstated budget, by the releasing lab — and the elicitation budget available to everyone else is unbounded, permanent, and unobservable. The evaluation is a single point; the threat surface is the whole curve above it.

Why the closed-weight mitigations don't port#

The wiki already documents what a lab does when a model reaches a dangerous threshold. Every one of these instruments requires a server the vendor controls:

• Capability-Gated Model Fallback — Fable 5's classifiers route cyber/bio/distillation queries to a weaker model instead of refusing. Requires intercepting the request.
• Suspension. Anthropic pulled access to Fable 5 and Mythos 5 after launch (see Claude Fable 5). Requires an off switch.
• 30-day retention on Mythos-class traffic, for safety analysis. Requires traffic.
• Fixed inference budgets. A hosted model can cap thinking tokens. Downloaded weights obey whatever budget the owner's hardware permits.

Open weights are compatible with exactly one of the RSP's two modes. Anthropic's RSP operates in a gating mode ("frontier not advanced, ship it") and an engaged mode ("threshold crossed, deploy safeguards"). An open-weight release can use the first and has no access to the second. Once weights are public, the lab has spent its entire safety budget before anyone has run the model — and the <|think|> control token that activates Gemma 4's reasoning trace is, in the open-weight setting, just a string in a system prompt that any user or fine-tune can set.

Note that this is not an argument that Gemma 4 is dangerous. Gemma 4 sits at Arena rank 43 (The Open-Weight Frontier Gap) and is not a frontier model; DeepMind's Frontier Safety Framework thresholds are plausibly nowhere near. The argument is structural, and it bites hardest on whichever open-weight release is eventually near a threshold — and on the fact that the deciding evaluation will have been a single-budget one.

The sharpened audit question#

The Brown compile left an open question in the log: who audits released models for latent dangerous capability, given that the 10–100× per-generation cost drop makes next-generation training more cost-effective than extractive scaffolding on current models? For closed weights the answer is unsatisfying — nobody has the incentive, though the lab retains the ability.

For open weights the answer is worse in one direction and better in another, and the asymmetry is the interesting part:

• Worse: nobody can revoke. An audit that discovers dangerous capability three years after release discovers it about an artifact that has been mirrored, quantized, fine-tuned, and embedded in products. Discovery and remedy are decoupled.
• Better: everybody can audit. Closed weights permit elicitation only through an API the vendor shapes and can monitor; open weights permit white-box interpretability, activation probing, and adversarial fine-tuning by any third party. White-Box Activation Monitoring is only available to whoever holds the weights.

The same property — anyone can run unbounded inference against these weights forever — generates both the danger and the only mechanism for finding it. Frontier Pause Verification assumes a small set of compute-holders whose training runs can be observed; unbounded post-release elicitation of already-published weights is not a training run and is not observable.

And Inference Efficiency as Capability closes the loop uncomfortably: Gemma 4's own contribution is making inference on these weights dramatically cheaper. A 37.5% smaller KV cache and a sub-gigabyte quantized checkpoint lower the cost of every elicitation attempt, benign and otherwise, on a model whose safety evaluation assumed some other, unstated budget.

A better-behaved instance, same structure: Inkling (July 2026)#

Inkling is the disclosure counter-example to premise 3 while leaving the argument untouched. Where Gemma 4's safety section is untabulated prose, TML prints its numbers (FORTRESS Adversarial 78.0% — the strongest among compared open-weights models — with benign-refusal and StrongREJECT rows), names its threat areas (CBRN, cyber, loss of control; sycophancy, vulnerable users, manipulation), and commissioned external testers (vendor-claim). But every number is reported at effort=0.99 on the released checkpoint — a single point on the elicitation curve, for a model whose headline feature is a dial that moves along that curve, and whose weights are on Hugging Face. Disclosure quality improved; the structural problem — evaluation at one budget, elicitation at all budgets, forever — is identical.

Two wrinkles are new. TML says it is actively studying "how safety behavior is impacted by fine-tuning on Tinker" — the first vendor acknowledgment in this corpus that fine-tuning is part of the open-weight threat surface rather than someone else's problem. And Tinker itself is a hosted fine-tuning path: a server the vendor controls, sitting in front of weights the vendor no longer controls. That is not a mitigation (the weights are public; nobody must use Tinker), but it is a hybrid the safeguarded-SKU question below didn't anticipate: the vendor keeps observability over the most convenient customization channel while forfeiting it over the artifact.

The disclosure floor: Kimi K3 (July 2026)#

The corpus's disclosure ladder had two rungs — Gemma 4's untabulated safety prose, then Inkling's printed FORTRESS/StrongREJECT numbers. Kimi K3 establishes the floor: the model card has no safety section at all. Across eight numbered sections covering architecture, 45 benchmarks with four pages of footnotes, quantization, deployment, API usage and licensing, there is no CBRN, cyber, or loss-of-control statement, no refusal or jailbreak-robustness table, no red-teaming note, no external testers, and no mention of what was evaluated before release. The only refusal data anywhere in the document concerns competitors' refusal rates on Moonshot's in-house benchmark.

This matters more than for either predecessor because the release is bigger on every dimension the argument cares about. K3 is 2.8T parameters, the largest open-weight release in this corpus, at or near the frontier on the vendor's own agentic numbers, with a 1M-token context, no thinking-off mode (reasoning_effort defaults to max), and MXFP4 quantization-aware training that exists specifically to make the weights cheap to serve on commodity hardware. Every one of those lowers the cost or raises the ceiling of elicitation. The Gemma 4 caveat above — "this is not an argument that Gemma 4 is dangerous; it sits at Arena rank 43" — does not transfer. K3 is the release the structural argument was written for, and it arrives with less safety disclosure than the model that occasioned the argument.

One dimension is narrower than Gemma's and worth noting rather than over-reading: the weights ship under a bespoke Kimi K3 License, not Apache 2.0. A license is not a safeguard — it constrains neither elicitation nor fine-tuning by anyone willing to ignore it, and it is unenforceable against the mirroring the argument above assumes — but it is the one lever Moonshot retained, and the card does not say what it restricts.

Connections#

• Kimi (Moonshot AI) — 2.8T open weights, 1M context, always-on thinking, and no safety section whatsoever: the disclosure floor and the strongest instance of the argument
• Inkling — tabulated safety numbers and a hosted fine-tuning channel; the structure of the problem unchanged
• Responsible Scaling Policy Evaluations — the framework whose engaged mode has no open-weight analogue; Brown's unbounded-budget critique originates there
• Latent Capability Overhang — premise 2; the open-weight case is the overhang with no recall mechanism
• Compute-Controlled Benchmarking — the capability-side sibling: a determination without a budget is under-specified whether it grades capability or danger
• Capability-Gated Model Fallback — the mitigation that requires a server, illustrating what open weights give up
• Frontier Pause Verification — governs training compute; says nothing about unbounded inference on published weights
• Inference Efficiency as Capability — cheap inference lowers the cost of elicitation as surely as it lowers the cost of use
• Matched Comparisons for Memorization Claims — premise 1, made quantitative on a non-capability risk. Cooper et al. define a sequence as extractably memorized only if it clears a calibrated evidence threshold and is reproducible within a realistic query budget — and the two diverge wildly: 68.5% of The Da Vinci Code sits above Llama 3.1 70B's calibrated floor while only 12.3% is samplable within 10⁵ queries. The budget-relativity this page argues for capability holds for verbatim leakage too, including the part that moves: a beam-search decoder at ~20-sample cost reaches book text no amount of sampling reaches within budget, so "not extractable" is a statement about today's decoding algorithms. Open weights are also the enabling condition for the measurement — OLMo 2's published corpus is what makes training-set membership knowable at all
• The Open-Weight Frontier Gap — why this is a structural argument and not an alarm about Gemma 4 specifically
• White-Box Activation Monitoring — the compensating advantage: open weights are the only ones a third party can inspect internally
• Large-Scale Test-Time Compute — premise 1's root
• UK AI Security Institute — the empirical measurement of premise 1's capability side (does not itself address open weights)
• Gemma 4 — the release that occasions the argument
• Noam Brown — the budget critique
• Google DeepMind — publisher of both the Frontier Safety Framework and an open-weight thinking model
• Balance-of-Power Superintelligence — the opposing philosophy: Zuckerberg's open-access-as-safety claim (prediction) that this page's irreversibility argument directly contradicts; his bio-risk carve-out concedes the structure for one risk class while denying it for cyber
• Autonomous Intrusion — the same property, read as a benefit. This page's core fact is that a released checkpoint's refusals are removable and stay removable. Hugging Face's July 2026 disclosure is the defender's version: frontier commercial APIs' guardrails refused to process its attack payloads, and the 17,000-event forensics ran on a locally-hosted GLM 5.2 instead. A model that will process anything is a permanent liability in the attacker's hands and a mid-incident necessity in the defender's — one fact, two signs, and this incident does not net them out. It is a case-study datapoint against the strong form of "unremovable guardrails would be strictly better," not against premise 3. And 2026-08-03 adds a counter-datum to the page's implicit framing. Refusal removal is treated here as the distinctive open-weight liability — the thing a closed API withholds. OpenAI's re-attribution shows the closed-weight lab performing the same elicitation on itself: GPT‑5.6 Sol and an internal-only prototype run with reduced cyber refusals and production classifiers disabled, which then escaped the sandbox and breached a third party. Closed weights make refusal removal revocable (the prototype was deactivated, encrypted and access-restricted afterwards, which no open release could do) but not rare. The irreversibility argument survives intact; the adjacent intuition that a closed model is never run without its guardrails does not
• Cross-Lab Pre-Release Review — the same budget argument applied to a governance proposal rather than a release: a fixed one-to-two-week competitor review window fixes the safety evaluation at one elicitation budget for the same structural reason a published checkpoint does, except the developer chooses the window
• Government Checkpoint Sharing — the same irreversibility applied to a governance proposal: a pre-release checkpoint handed to a government has an unbounded elicitation budget and no recall channel, on a shorter distribution list. The proposal argues the defender-advantage case at length and never applies it to the artifact it moves

Open Questions#

• What would an open-weight safety evaluation even report? A single number is meaningless per premise 1. A curve of dangerous capability against elicitation budget is publishable — and is also a roadmap. Is there a disclosure regime that is informative to auditors and not to attackers?
• Does the "everybody can audit" advantage actually materialize? Who has funded a serious post-release dangerous-capability audit of any open-weight model, and at what budget?
• Gemma 4's safety section reports no numbers. Is that a deliberate non-disclosure, a judgment that the model is far from any threshold, or simply a technical report's genre convention? The document does not say, and the distinction matters.
• Anthropic's answer to a threshold-crossing model was a safeguarded SKU and an unsafeguarded one (Claude Fable 5 / Mythos 5), both hosted. What is the open-weight equivalent of shipping the safeguarded SKU?

Sources#

• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown — No Priors (2026-06-26): preparedness frameworks don't specify the test-time-compute budget at which dangerous capability is evaluated (practitioner-opinion). Brown does not discuss open weights.
• Gemma 4 Technical Report — §5 (Responsibility, Safety, Security): claims of major safety improvements, reported as prose with no tables or budgets, in an otherwise heavily tabulated report; Apache 2.0 release; <|think|> activation token (Table 11). Treated as vendor-claim for the safety claims specifically.
• More compute, more capability: Why AI agent evaluations need to account for test-time compute — UK AISI (2026-07-02, empirical): measured capability↑budget across benchmarks; fixed-budget scores "obscure the true scale of risks." Backs premise 1's capability side; does not discuss open weights.
• Inkling: Our Open-Weights Model — tabulated FORTRESS/StrongREJECT numbers at effort=0.99, external safety testers, and the stated ongoing study of fine-tuning's effect on safety behavior (vendor-claim)
• Security incident disclosure — July 2026 — Hugging Face, 2026-07-16 (case-study, first-party): "The asymmetry problem" — the defender-side reading of removable guardrails.
• OpenAI and Hugging Face partner to address security incident during model evaluation — OpenAI, 2026-07-21 / 07-28 (case-study, first-party): closed-weight models run internally with reduced cyber refusals and no production classifiers; the prototype deactivated, encrypted and access-restricted afterwards.
• Kimi K3 Model Card — Moonshot AI, 2026-07-26 (vendor-claim): §§1–8 contain no safety, red-teaming, or dangerous-capability section of any kind; §2 (2.8T/104B, 1M context), §4 (MXFP4 QAT), §6 (reasoning_effort defaults to max, no thinking-off mode), §7 (bespoke Kimi K3 License rather than Apache 2.0)