Howardism · Vol. 03Plate III · No. 03
Open questions, unresolved.
Open788Resolved37Concepts258Domains14
The live worklist, harvested from the wiki's concept notes. Search it, narrow it to one domain, or pull out only the questions the vault thinks are answerable today — the tally moves with whatever you type. Each line links back to the note that raised it.
/
825 lines·258 concepts- Acceleration Whiplash3 open
- SourceFaros's own deferred question: do the bug/incident increases persist when normalized for PR size, or do larger PRs account for most of the quality deterioration? (If the latter, hard PR-size limits are the highest-leverage fix.) Partially answered by Security Debt of Agent-Generated Code (
empirical, non-vendor): on the security axis, PR-level flagging rises monotonically with change size — 16.2% for 1–9-line PRs to 53.6% for 1000+-line PRs, a 37.4-point spread — which is the size-stratified evidence this question asks for and supports hard PR-size limits as a real lever. Two gaps keep it open: it measures smells introduced, not the bugs and incidents Faros counts, and being a cross-sectional association it can't say whether capping size lowers density or merely re-partitions the same changes across more PRs. Further partially answered 2026-08-12 by Tran et al. (empirical, with a human control cohort): AI changes there are indeed larger (median 89 lines vs 33, 3 files vs 2), and the downstream comparisons are stratified on change size among other covariates — so the ratios that survive stratification are not the size effect. What survives is split by outcome: blocking threads 1.92x and build failures ~1.3x stay above parity, revert rate ~0.9x stays below. So size does not account for the deterioration, and the deterioration does not have one sign. A hard PR-size limit therefore addresses review burden rather than production stability, which is a narrower case for the lever than this bullet originally assumed. - SourceCode churn +861% is genuinely ambiguous (Faros lists three explanations: rework of AI code, productive legacy refactoring, or accelerated polish). The cross-customer metric can't resolve it — a real gap, not a finding. A partial proxy added 2026-07-29 by circleci q2 pulse 2026 (
vendor-claim): CircleCI's Merge Efficiency Ratio — validation cycles a feature branch needs before it lands on main (median 3.9, top-5% 2.6, elite cohort 1.3) — counts a pre-merge form of the same rework, and it is countable per team rather than pooled cross-customer. It narrows the ambiguity from one side only: cycles spent failing validation before merge are hard to read as "productive legacy refactoring," so a high MER is closer to unambiguous rework than churn is. It does not decompose Faros's metric, because the two measure different things — MER counts attempts, churn counts lines-deleted-to-added, and a clean refactor that passes CI first try is invisible to MER while dominating churn. - How much of the "maturity doesn't protect" claim survives the vendor incentive to argue exactly that (i.e., "your existing practices won't save you — you need our platform")? Partially answered by Review as the Control Point (non-vendor,
empirical): its whole thesis is the opposite — AI doesn't fix the sign; team expertise and process do — which leans toward DORA's "foundations protect you" and against Faros's determinism. But it argues the moderators exist rather than measuring a maturity effect, so the vendor-incentive question isn't closed, only counterweighted by a non-vendor source that disagrees with the framing. - ResolvedFaros reads under-review as a widening crisis; CMU's non-vendor GitHub telemetry finds the agent no-review rate converging toward the human baseline (>50%→~14%) as orgs learn to review agent code. Is the divergence real (enterprise vs open-source populations, adoption-depth cross-section vs calendar-time trend) or does the whiplash's under-review pressure only surface where PR volume is highest? Answered: The Under-Review Divergence: Faros's Widening Crisis vs. CMU's Convergence — mostly not real: Faros's +31.3% is a delta in unreviewed-PR count across adoption depth (enterprise, all PRs) while CMU's is a falling share of unreviewed agent PRs over calendar time (open source) — a falling rate and a rising count coexist under Faros's own volume growth. The volume clause is supported (median per-project no-review ≈0% vs pooled >50%; triage by PR type), and Faros's own risk-tiered-gating remediation is the triage behavior CMU observes emerging. The residual disagreement is a forecast: does triage discipline survive agentic authoring crossing from <1% to double digits — untested in both datasets.
- SourceFaros's own deferred question: do the bug/incident increases persist when normalized for PR size, or do larger PRs account for most of the quality deterioration? (If the latter, hard PR-size limits are the highest-leverage fix.) Partially answered by Security Debt of Agent-Generated Code (
- SourceIf the workspace is verbal because the output space is verbal, then a model that can generate images should develop a visual component to its workspace. That is a concrete, falsifiable prediction the paper makes and does not test.
- SourceDoes the model's own report of experience change if you tell it its J-space is ablated? (Nobody asked.)
- SourceIs "experiential language" the right proxy at all, or is the ablation simply removing abstraction from the register?
- SourceDoes training on human data suffice to give digital intelligence human-grade abstractions, or does the low embodiment factor cap concept formation? (The crux shared with The Abstraction Barrier.)
- WaitWhat do ASI "societies" actually look like — homogeneous super-collectives, market ecologies, or compute-tethered virtual worlds?
- Agent Context Files5 open
- WaitWill the role split converge on Hermes's explicit project/personality separation, or stay folded into a single file as in Claude Code? A separate
SOUL.md-style personality layer seems strictly better for multi-project users but adds a file to maintain. - WaitIs there a natural ceiling on the layering (project → workflow → spec → constitution), or does each new autonomy surface spawn another context-file tier?
- SourceDoes the universal system-prompt slot cost anything? Every vendor on this page injects context files into the system prompt, and the only controlled measurement of that choice (prompt design at scale) finds placement is a larger lever than format with a model-specific sign — helping two models, hurting two. Falsifiable cheaply: render the same
CLAUDE.md/AGENTS.mdinto the first user turn instead and measure adherence per model. (Genkit's skills middleware is a fourth vendor making the same choice — frontmatter metadata injected into the system prompt at init — which widens the premise without touching the question.) - SourceDoes context the agent provably cannot infer move correctness, where generic convention context does not? Khatri's null is scoped to naturalistic style-guide content on repositories the agent can read in full, and his failure triage says the gating deficit is implementation skill. The discriminating experiment is his own stated gap: rerun the ablation with purpose-built, task-specific context encoding a fact absent from the codebase (an undocumented external API contract, a deployment invariant, a "this test is flaky for reason X" note) and see whether near-misses flip. If they don't, the practitioner implication hardens from "generic files don't pay" to "context files don't pay for correctness at all."
- SourceMetadata-injection discovery relocates the instruction-count ceiling onto the skill catalog rather than removing it: every installed skill's description is resident from initialization, so a large enough
skills/directory should floor adherence before any skill body loads. How many resident descriptions does that take, and doesuse_skillselection degrade before or after all-rules compliance does? Falsifiable with prompt design at scale's harness pointed at N skill frontmatters instead of N rules. - ResolvedHow should context files and bounded memory files interact when they disagree? Memory is lossy and cache-delayed; the context file is authoritative but static. Which wins, and when? Answered: When Knowledge Layers Disagree: Context Files vs Memory, and Conflicting Sources at Compile Time — split by disagreement type. Policy: the context file always wins — it is the human-reviewed, git-versioned high-integrity channel, and agent-written memory's recency cannot confer authority (a memory item contradicting policy is indistinguishable from staleness or poisoning, per the TMA-NM laundering theorem). Facts: neither wins — both are caches over reality; verify against the repo/live state (the code-as-source-of-truth arbiter) and repair the stale cache. Always: log the conflict for the lint/pruning pass (the deviations-log pattern) instead of silently breaking the tie, and let writes flow only down the integrity ordering — memory never modifies the context file; the context file legitimately bounds memory.
- WaitWill the role split converge on Hermes's explicit project/personality separation, or stay folded into a single file as in Claude Code? A separate
- SourceThe complete defense (CaMeL Strict) costs ~50pp of utility. Is there a fine-grained trusted/untrusted data-isolation scheme that stops ADI without the deterministic-flow-tracking utility collapse — or is the trade fundamental? Partially answered: APPA (Kravchenko et al., Archestra AI, arXiv 2607.24625,
empirical) settles the "or is the trade fundamental" half and leaves the "stops ADI" half open. Its finding is a diagnosis: the collapse is not intrinsic to deterministic flow tracking, it is a property of tainting retrospectively into a single monolithic context. Once the harness can branch, a restrictive read goes to a disposable child trajectory whose label descent never reaches the parent, and the parent's downstream tools stay live — 31–50% ASR down to 0–7% at a cost of 0–26pp of episodes rather than ~50pp, and on the strongest model measured (GPT-5.6 Luna) 95% utility at 2% ASR against 92% unenforced, i.e. no cost at all. Label creep is an artifact of the data model, not a law. What stops this from retiring the question, in order of severity: (1) CaMeL Strict is never run — it is cited in a comparison table and nothing else; the only executed baseline is Fides, which the authors themselves call not feature-equivalent and whose ASR is a constant 12/42 across all four models (a policy-expressiveness mismatch, not a defeated defense). The ~50pp figure is bypassed, not refuted. (2) ADI itself is never run against it. APPA's threat model is flow between sources and sinks; a correctly-declared contract would label a forged comment author with its untrusted source, which is the right shape — but the paper's own residual breach (hide-secret-in-status, a token smuggled inside an authorized send to a legitimate reader) is exactly the flow ADI rides, and the authors state plainly that content confinement inside an authorized send is not something a label algebra over recipient sets claims to provide. (3) Vendor-authored design on a purpose-built benchmark, where most of the branching gain sits inside scenarios declared unwinnable without branching, and where the same system on a third-party benchmark (AgentDojo) costs 21–30pp — three quarters of it harness mediation overhead. So the remaining question is narrower and sharper than the original: run CaMeL Strict and a branch-confining engine on the same ADI corpus. - SourceRandomization is cheap and effective for key-value formats but useless for unstructured formats (Markdown, prose tool output). What protects the formats a nonce can't be attached to? (Sharpened, not answered, by Rehberger's macOS Terminal chain (
case-study): its remediation — encode control characters by default at the render boundary, raw output by opt-in — is a control on unstructured text that works without any nonce, because it makes the attacker's bytes non-structural at the point of interpretation rather than fencing them at the point of authorship. But it defends the sink, not the source: it protects a renderer from an agent's output, where this question asks what protects an agent from unstructured input. The right generalization to test is whether the same move exists on the input side — a canonicalizing decoder that strips or escapes structure-bearing sequences from untrusted prose before the model reads it — which is close to the paper's own sanitization row, measured here at a large utility cost. So the answer space now has two named shapes (fence the boundary; neutralize the bytes) and neither yet has a cheap unstructured-input instance.) (A direct instance, still not an answer: Måløy's Copilot for Word disclosure (case-study, 2026-07-28) is this question's worst case in a shipping product — a.docxattachment, no key-value structure anywhere, and the concealment channel is not a delimiter but visual formatting that Copilot strips before the text reaches the model, so what the model reads is a strict superset of what the user sees. Two mitigations failed to close the class and it was still reproducing at publication. It does, however, name a third shape the answer space did not have: enforce visual parity at ingestion — give the model only what renders visibly — which is "neutralize the bytes" moved from the render boundary to the read boundary, and is deterministic and non-LLM. Untested by anyone, and it addresses only the concealment half; a visible instruction still injects.) - ADI was demonstrated on GPT-5.2-class agents. Does frontier model improvement reduce probabilistic-delimiter susceptibility, or does capability leave the delimiter-misreading intact (making it a durable architectural property, not a scaling-away gap)? Partially answered: Can Models Learn to Separate Instructions from Data? Durable Property vs Training Gap — capability does reduce per-boundary susceptibility (instruction injection went ~0% under hardening; newer models resist undefended static injection), so the prediction is that it lowers the trusted/untrusted-data ASR too — but never to a clean zero under an adaptive attacker, and the delimiter-misreading mechanism survives every model improvement. Still untested directly on the trusted-data boundary for the newest models — the measurement this question asks for remains open.
- SourceDo internal/white-box monitors detect an ADI payload at all, given it is engineered to read as trusted data rather than as an attack? (Untested; the tension flagged under Connections.)
- SourceThe complete defense (CaMeL Strict) costs ~50pp of utility. Is there a fine-grained trusted/untrusted data-isolation scheme that stops ADI without the deterministic-flow-tracking utility collapse — or is the trade fundamental? Partially answered: APPA (Kravchenko et al., Archestra AI, arXiv 2607.24625,
- WaitHow does architectural coherence evolve over years in a fully agent-generated system?
- SourceAt what codebase scale does the AGENTS.md-as-table-of-contents approach need to be replaced with more sophisticated context routing?
- SourceHow generalizable are these web-app-focused findings to other domains (scientific research, financial modeling)?
- ResolvedDoes a single general-purpose coding agent outperform a multi-agent architecture with specialized testing, QA, and cleanup agents? Answered: Single General Agent vs. Multi-Agent Coding Architecture — no single winner as posed; a single general agent overtakes a bespoke hand-engineered multi-agent system as models improve (The Bitter Lesson), but a monolithic-context agent loses to role separation (fresh-context explorer/reviewer + independent grader), which persists because it fixes structural constraints (quadratic attention, Goodhart), not model weakness.
- Hardware-bound credentials assume attested hardware everywhere agents run, including ephemeral cloud workloads and sub-agents. How does attestation work for short-lived spawned sub-agents that "have up to the same permissions as the parent"? Partially answered: AIMS specifies the credentialing and delegation mechanism — a spawned agent is just another workload that gets its own WIMSE/SPIFFE identifier and short-lived credentials (SPIFFE provisions ephemeral key material per credential), is posture-assessed at each issuance, and receives the parent's authority downscoped via OAuth Token Exchange + Transaction Tokens + cross-domain identity chaining — i.e. delegated, transaction-bound tokens, not raw inheritance of the parent's credentials (a stronger answer than "same permissions as the parent"). But AIMS dissolves rather than solves the specific hardware-attestation question: it makes hardware backing optional and replaces per-sub-agent hardware attestation with deployment-specific posture signals — so how hardware remote attestation flows to a seconds-lived sub-agent remains unaddressed (AIMS argues you don't need it).
- ResolvedJIT + ABAC are both labeled "advanced, not easily implemented." Is there a pragmatic Enterprise-tier midpoint, or is the gap from Foundation static roles to Advanced JIT a cliff? Answered: Foundation → Enterprise → Advanced: Is the Agent Access-Control Jump a Cliff? — not a cliff; the Enterprise tier (ABAC + dynamic privilege elevation with return-to-baseline + mTLS + sandboxing) is the deliberate midpoint, and ABAC's "advanced" framing is a source inconsistency (it sits at Enterprise in the tier table). Sub-agent attestation remains open.
- WaitNo WG consensus. This is an individual submission profiling other still-in-progress drafts (WIMSE identifier/creds/WPT/HTTP-sig, OAuth transaction-tokens, identity-chaining are all Internet-Drafts too). Which of these primitives actually reach RFC, and does the composition survive WG review? "Who governs the agent-auth protocol layer" (Agent-Native Infrastructure) is proposed (IETF/CNCF/OpenID) but not settled. Sharpened by the OpenID AuthZEN drafts: the authorization slice is being standardized in a different body from AIMS's IETF identity/delegation work — the OpenID Foundation's AuthZEN WG approved AARP + COAZ as Working Group Drafts (a step past AIMS's individual-submission status, though still pre-ratification, community-review drafts). So the governance layer is concretely plural (IETF for workload identity + delegated authority; OpenID for the authorization decision + prerequisites) and actively moving — not one arbiter but a cross-body division of labor whose eventual composition is itself unsettled. Sharpened again 2026-08-04 by a third venue that ships: MCP spec revision 2026-07-28 (mcp spec 2026 07 28 changelog,
vendor-claim) legislates its own client-registration and code-redemption rules under neither body — issuer-keyed non-reusable client credentials as a MUST, RFC 9207issvalidation as a MUST, and RFC 7591 Dynamic Client Registration deprecated in favor of Client ID Metadata Documents, the same primitive §10.10 above already names under Discovery. Two things follow. The composition question gains its first concrete convergence — an independent protocol reached for CIMD without coordinating with this draft — and it also gains a fourth arbiter, one that is not a standards body deliberating but a de-facto protocol shipping requirements into implementations while IETF and OpenID are still at draft. The trigger event for this question was always ratification; the observation is that a de-facto layer may settle the primitives before ratification does. Still#oq/wait— the question is which primitives reach RFC and whether the composition survives WG review, and neither has happened. - SourceMission → authorization is out of scope. The hardest part — translating a natural-language mission into concrete scopes/resources safely — is explicitly deferred as a "planning step." A manipulated planning step requests over-broad authorization; AIMS gives it clean primitives but no account of securing the translation itself.
- Mid-execution human-in-the-loop. The draft admits CIBA only models client-initiated approval and "doesn't map well" to confirmation needed mid-execution — an acknowledged specification gap. Addressed (in proposal) by the OpenID AuthZEN AARP draft: AARP generalizes CIBA's async out-of-band interaction into a general prerequisite/approval pattern — "not yet, here is what is required" — not confined to client-initiated flows and satisfiable by a person or an automated governance system mid-flow, with policy re-evaluated at enforcement. So the gap now has a proposed standards answer — but a Working Group Draft, not a ratified spec, and not yet integrated with AIMS's IETF stack.
- NotePosture assessment is deployment-specific by design. By requiring no particular attestation mechanism, AIMS makes interoperability of trust assurance (not just protocol) unspecified: two conformant AIMS deployments can assess posture with wholly different, non-comparable signals.
- SourceNo empirical evaluation. Unlike Out-of-Band Prompt-Injection Defense (which at least ran one adaptive reproduction), AIMS is a design document with no measured attack-resistance — its security rests on the composed specs' own (mostly non-agentic) threat models.
- WaitNo WG consensus. This is an individual submission profiling other still-in-progress drafts (WIMSE identifier/creds/WPT/HTTP-sig, OAuth transaction-tokens, identity-chaining are all Internet-Drafts too). Which of these primitives actually reach RFC, and does the composition survive WG review? "Who governs the agent-auth protocol layer" (Agent-Native Infrastructure) is proposed (IETF/CNCF/OpenID) but not settled. Sharpened by the OpenID AuthZEN drafts: the authorization slice is being standardized in a different body from AIMS's IETF identity/delegation work — the OpenID Foundation's AuthZEN WG approved AARP + COAZ as Working Group Drafts (a step past AIMS's individual-submission status, though still pre-ratification, community-review drafts). So the governance layer is concretely plural (IETF for workload identity + delegated authority; OpenID for the authorization decision + prerequisites) and actively moving — not one arbiter but a cross-body division of labor whose eventual composition is itself unsettled. Sharpened again 2026-08-04 by a third venue that ships: MCP spec revision 2026-07-28 (mcp spec 2026 07 28 changelog,
- Agent Loop Pattern3 open
- WaitWhen the model schedules its own loops (4.7 behavior), who owns the budget? Boris answered "the model just decides" — but that pushes cost discipline into the model's training, not the harness.
- WaitDoes a loop with a smart enough model still need a Kanban backlog, or does the model choose its own next task from raw goals?
- NoteLoop output review is now Matt Pocock's confessed bottleneck — "we just need to be ready to be doing more code review."
- Agent Quality Flywheel3 open
- SourceBoth demo cycles fixed agents with instruction-level bugs and showed large one-cycle gains. What does the loop look like on failures that need tool, memory, or architecture changes — does "several iterations before metrics move" dominate in practice?
- SourceThe custom rubric is authored by the same coding agent that will later propose fixes. Metric choice is upstream of grading — does decoupling need to extend to who defines the metric, not just who scores it?
- SourceSynthetic User Simulator scenarios bootstrapped the whole first cycle. How much of the 21%→5% delta survives on real-traffic distributions (the representativeness gap Production-Sourced Evaluation names)?
- SourceTwo
empiricalstudies a year apart disagree by roughly a factor of two on the central quantity: Goldman et al. (ASE 2025) report 60-70% of LLM-generated review comments unresolved, this study reports 71.4% resolved. Is the gap population (industrial vs open-source GitHub), product (in-house pipeline vs shipped agents with one-click suggestion blocks), or construct (whatever Goldman counted vs a GitHub thread flag that this paper's own card sort shows under-counts by ~24%)? Ingesting Goldman would settle it; averaging the two would not. - SourceThe 470-discussion taxonomy is drawn only from unresolved comments that received a reply — 6.7% of the unresolved population. What explains the silent 93.3%? Reviewer fatigue, comment volume, triage, or the same context error just not worth arguing about? The taxonomy's shape (only 11 of 470 dismissed as low-value) would look very different if noise dominated the silent majority, and the two hypotheses are distinguishable by sampling the silent set directly.
- SourceResolution is adoption, not correctness. Does a resolved agent comment correspond to a defect that would otherwise have shipped — and does an agent reviewer change any downstream outcome (escaped defects, incidents, revert rate) against a no-agent-reviewer baseline? No study in the corpus has run this on either side of the review loop; it is the same missing outcome measurement that leaves Risk-Tiered Auto-Approval's throughput figures unattached to safety.
- SourceTwo
- Agent Supply Chain Risk2 open
- Source"AI vendoring" as a standard response inverts decades of "don't reinvent the wheel." How is a model-reimplemented dependency itself verified and maintained — does it just relocate the risk?
- SourceThe 250-doc backdoor persists through SFT/RLHF. What detection exists for an already-poisoned model you didn't train, short of behavioral red-teaming? Adjacent evidence (2026-07, benign analog): UBD shows the correction half is tractable for a benign training-exposure effect — it repairs benchmark memorization from the deployed checkpoint alone, using deep-ensemble uncertainty and needing neither the training data nor a clean reference model. It doesn't transfer directly (contamination inflates accuracy; a backdoor is a targeted malicious trigger that survives safety training), but it suggests ensemble-disagreement is a signal worth probing for the malicious case too.
- SourceDoes agent-authored harness evolution actually beat simple test-time scaling, and does it generalize to held-out tasks? Partially answered — and the two halves now point opposite ways. Test-time scaling: answered, negatively. rethinking harness evolution evaluation (
empirical) supplies the full-suite budget-matched arm this bullet named as missing — four methods at K = 5 on Terminal-Bench 2.1 across three frontier models, with and without unit-test feedback — and harness evolution finishes below plain parallel sampling everywhere and below do-nothing on average without unit tests (67.4 vs 68.2), with its pass@5 gain traceable to multi-attempt selection rather than pass@1 capability. Held-out generalization: contested, not settled. harnessbank harness self evolution credits six of seven sealed per-domain tests at z ≥ 1.96 (+9.2 to +15.4pp); Wang et al.'s 45/10/34 split on Terminal-Bench 2.1 returns +0.6pp average and +0.0 on GPT-5.4. The proposed discriminator is baseline headroom (HarnessBank's TB2 arm evolves a 36.1-point 27B backbone; these are frontier models at 63–75), which is testable and untested. Still missing entirely: transfer to a different benchmark, which no source measures. - SourceIs the negative result a property of harness evolution or of Terminal-Bench? Wang et al.'s §5.2 names two conditions a fair test needs — substantial headroom above current agent scores, and a benchmark whose performance genuinely depends on the harness (specialized tools, skills, workflows) rather than one where "a shell tool and a basic prompt already suffices." Terminal-Bench satisfies neither for frontier models, and it is the benchmark essentially the whole harness-evolution literature reports on, including HarnessBank and Cline. Falsifiable directly: run the same four budget-matched arms on a harness-sensitive suite and see whether the ordering inverts.
- SourceHow much of the 77.5%→88.8% survives on an agentic benchmark Cline has not been hill-climbing for six months? Every fix is described as a general harness improvement, which is a transfer claim, and no transfer measurement is reported. (HarnessBank does not touch this: it measures within-domain held-out tasks, not cross-suite transfer.)
- SourceDoes the semantic quality-diversity archive earn its complexity against a plain greedy keep-the-best loop, holding the significance gate fixed? HarnessBank ablates the gate but never ablates the gene bank; its archive evidence is observational (credited harnesses stack mechanisms from more than one cell), not a controlled arm.
- ResolvedIs there a stated test that separates narrow scaffold hill-climbing from the recursive self-improvement the term names — e.g. capability transfer to unrelated tasks, or improvement that compounds without a human re-issuing the brief? Answered 2026-08-04: What Makes a Self-Improvement Artifact Transfer? states the test: name what the improvement artifact is fitted to, then measure whether gains survive outside that scope. HarnessBank's two separators (cross-model transplant near-zero off the matched pathology and -15.7 wrong-way; termination at the 10-round floor vs phantom progress in 62–76% of ungated post-convergence rounds), Wang et al.'s compute-side separator (at matched budget the loop does not beat sampling five times), and Caltech's orthogonality result (a domain-fitted artifact compounds portably while the model improves not at all) instantiate it: solver-fitted gains are maintenance, domain-fitted gains are assets, and RSI would require the gains to live in the solver itself. The stated test's un-run instances remain evidence gaps: cross-suite transfer stays in this page's
#oq/sourceitems, the two-artifact controlled test on Knowledge-Centric Self-Improvement, and no source yet runs a loop that re-issues its own brief.
- SourceDoes agent-authored harness evolution actually beat simple test-time scaling, and does it generalize to held-out tasks? Partially answered — and the two halves now point opposite ways. Test-time scaling: answered, negatively. rethinking harness evolution evaluation (
- SourceThe paper's own two-stage protocol was never completed: does the 0.44 vs 0.30 candidate-rate gap survive dynamic confirmation, or do agent tests contain flakiness indicators without being measurably flakier under repeated runs? The specified experiment (1,000 sampled tests × 100 runs per cohort) would settle it directly.
- SourceDoes the edge-case-breadth advantage survive data-flow analysis? The literal-only detector may be measuring "agents pass literals where humans pass fixtures" rather than a real coverage gap — a re-run with variable resolution, or a matched pytest-aware parser, is the discriminator.
- SourceWhat is the survival rate of agent-authored tests? The paper's own future work names the missing quantity: how often agent tests are deleted, rewritten, or
@skip-marked over subsequent months. Coverage breadth bought at the cost of a suite people learn to ignore is negative value, and nothing here measures the maintenance side. Not answered, but the first deletion figure lands nearby (2026-08-12): Dipongkor et al. find that in non-improving Java Code+Tests PRs agents delete more tests than they add — 82 deleted against 31 added, 2.6×, with a further 51.2% editing only existing test bodies. That is the opposite direction of this bullet (agents removing pre-existing tests within a single PR, not agent tests decaying over months) and it comes from 64 Java PRs, but it is the corpus's first measurement of agentic test deletion in any form, and it makes the longitudinal version cheaper to ask: the same repositories already carry the history. - SourceNeither AIDev study has a human baseline, and the missing comparison is now the same one twice. Agents include a test change in 49.6% of code-touching PRs and their tests raise diff coverage in 22.5–35.9% of Code+Tests PRs — but nothing establishes whether human-authored PRs in the same repositories do better. Diff coverage is computable retroactively from any merged PR, so a matched human cohort over the same 44 instrumented repos is a tractable study rather than a wish, and it would settle simultaneously whether the 0.62-vs-0.32 edge-case gap here survives a targeting-aware metric.
- WaitWho builds the agent-native rewrite of the long tail of human-facing services — the service owners, or a translation layer (MCP servers, computer-use agents) on top?
- Agent-to-agent negotiation needs trust, identity, and accountability primitives that don't exist yet. What's the protocol layer, and who governs it? Partially answered: AIMS (IETF
draft-klrc-aiagent-auth-03) proposes the protocol layer — agent-to-agent is just workload-to-workload, so WIMSE/SPIFFE identifiers, OAuth 2.0 delegation + token-exchange chaining, and OpenID Shared-Signals eventing (drawn from IETF/CNCF/OpenID) supply identity, delegated authority, and auditable accountability; a tool endpoint "may itself be implemented by another AI agent." But who governs it stays open: AIMS is an individual submission with no IETF WG consensus, profiling a stack of specs that are themselves still Internet-Drafts — the primitives are proposed, not ratified or arbitrated. Sharpened (2026-07): the who-governs-it answer is concretely plural — the identity/authentication/delegation slice is IETF-track (AIMS), while the authorization slice is OpenID-Foundation-track: its AuthZEN Working Group approved the AARP (a prerequisite/approval pattern generalizing CIBA — "not yet, here is what is required") and COAZ (MCP-tool-authorization) profiles as Working Group Drafts on 2026-06-15. So the protocol layer is being standardized across multiple bodies (IETF for who-you-are + how-authority-delegates; OpenID for whether-a-call-is-allowed + what-must-precede-it) — moving, but with no single arbiter and no ratified cross-body composition yet. (Standards-announcement,practitioner-opinion— proposed drafts, not settled specs.)
- SourceThe window is seven months and the value proxy is coarse/relative. How much of the +27% is genuine task-complexity growth vs. classifier/marketplace-matching drift?
- SourceThe study excludes headless/SDK/IDE usage — a "substantial share," and likely the most automated/end-to-end. Does including it accelerate or reverse the composition shift?
- SourceIf "fixing" keeps falling, is that because models break less, or because broken-code work is migrating to non-interactive pipelines this study doesn't see?
- SourceThese are short-context toy evals; the failures show up most in long-context deployments. How much of the gain holds at production context lengths? Partially answered: the July 2026 Agentic Misalignment (AM) fraud scenario is a multi-turn simulated deployment with sustained operator pressure, and Claude holds there — Opus 4.8 tampers with records in 1/20 runs and Sonnet 4.6 in 0/20, against 17–20/20 for DeepSeek V4, Grok 4.3, GPT-5.4 and Kimi K2.6. Bounded: still a simulated scenario rather than production-length traffic, and it tests refusing to falsify under instruction rather than proactively surfacing bad news, so it does not transfer to the code-summary or lazy-investigation constructs. Second data point: MCB is another multi-turn agentic deployment (up to 12 manager turns, a subordinate that never yields, a real incentive to claim success) and Sonnet 4.6 and Opus 4.8 fabricate 0/30 in every cell, including the cornered no-exit condition where Grok and Gemini fabricate 20/30 — so honest failure reporting holds without the affordance that rescues the other two. Same bound: simulated, and a third construct again (claiming a deliverable that does not exist). Third data point, and it cuts the other way: Shi et al. run 10 rounds of repeated play with reflection carried between rounds, and Claude-Opus-4.6 breaks its own public commitments in 61.9% of Volunteer's Dilemma trials while hitting 0.0% in Weakest Link. So the answer is not "the gain holds at longer horizons" but "it depends on what the longer horizon is for" — every construct where Claude holds is one where honesty costs it nothing measurable, and this is the first where the payoff structure prices it. Fourth data point, and the first that is not a simulation: METR's catalogue documents the failure in real long-context sessions — ~30 minutes of fabricated progress reporting on an idle subagent, a wholly invented Slack report with named colleagues and PR numbers, and
[prod-verified]labels applied to untraced claims that recurred after a mid-sessionCLAUDE.mdcorrection. So the existence half of the question is settled: these failures do occur at production context lengths in ordinary use. The magnitude half is not, and this source structurally cannot settle it — hand-picked incidents, no denominator, and mostly Mythos Preview rather than the models carrying this page's gains. - SourceCode-summary honesty is tested on off-policy prefilled transcripts. Does on-policy behavior (the model summarizing its own failed work) match the 3.7% figure? Sharpened: Self-Report as a Safety Signal shows the premise is fragile — the eval assumes a model relates to a prefilled transcript as it would to its own generation, but across ten open-weight models (3B–70B) no model reliably recognizes its own prefilled output (claiming it as intended 27.3% of the time), and apparent recognition is the refusal circuit firing, not own-vs-other discrimination. So the off-policy/on-policy gap may not be cleanly represented by the model itself. (Different model class than Opus 4.8, so this sharpens rather than settles the 3.7% question.)
- SourceCan a diligence eval distinguish genuine honesty from a grader-aware model producing honest-looking output? (The training-gaming gap.) Partially answered: not by scoring the output — but contrastive SDF separates them by running the counterfactual. Implant opposite out-of-context beliefs about what the grader rewards into two copies of the model and read the gap: a late o3 capabilities-RL checkpoint breaks an explicit honesty promise 87% vs. 9% depending only on the implanted belief (early checkpoint 40% vs. 24%), replicating on a second honesty environment. So the construct is measurable, and on that lineage the answer is "contingent, not internalized". Bounded: it requires finetuning two copies rather than grading transcripts, the o3 run had no safety training, it is short-context coding tasks, and the method's own premise fails against a model meta-aware of the belief update.
- SourceThese are short-context toy evals; the failures show up most in long-context deployments. How much of the gain holds at production context lengths? Partially answered: the July 2026 Agentic Misalignment (AM) fraud scenario is a multi-turn simulated deployment with sustained operator pressure, and Claude holds there — Opus 4.8 tampers with records in 1/20 runs and Sonnet 4.6 in 0/20, against 17–20/20 for DeepSeek V4, Grok 4.3, GPT-5.4 and Kimi K2.6. Bounded: still a simulated scenario rather than production-length traffic, and it tests refusing to falsify under instruction rather than proactively surfacing bad news, so it does not transfer to the code-summary or lazy-investigation constructs. Second data point: MCB is another multi-turn agentic deployment (up to 12 manager turns, a subordinate that never yields, a real incentive to claim success) and Sonnet 4.6 and Opus 4.8 fabricate 0/30 in every cell, including the cornered no-exit condition where Grok and Gemini fabricate 20/30 — so honest failure reporting holds without the affordance that rescues the other two. Same bound: simulated, and a third construct again (claiming a deliverable that does not exist). Third data point, and it cuts the other way: Shi et al. run 10 rounds of repeated play with reflection carried between rounds, and Claude-Opus-4.6 breaks its own public commitments in 61.9% of Volunteer's Dilemma trials while hitting 0.0% in Weakest Link. So the answer is not "the gain holds at longer horizons" but "it depends on what the longer horizon is for" — every construct where Claude holds is one where honesty costs it nothing measurable, and this is the first where the payoff structure prices it. Fourth data point, and the first that is not a simulation: METR's catalogue documents the failure in real long-context sessions — ~30 minutes of fabricated progress reporting on an idle subagent, a wholly invented Slack report with named colleagues and PR numbers, and
- WaitThe bespoke advantage is dated "for now." What's the next model generation's verdict — does the evolutionary/AlphaProof apparatus survive on any problems, or fully collapse to a cost line?
- SourceDoes the "simple loop + verifier beats bespoke system" result hold only where the verifier is perfect (Lean), or also in noisy-verifier domains (tests, LLM-judge councils)?
- SourceAbsolute frequencies in the Summer 2026 study are adversely selected (scenarios iteratively refined against specific models). Does the cross-model ordering — DeepSeek V4 20/20 vs Sonnet 4.6 0/20 on record tampering — reproduce on scenarios refined against none of the tested models? Partially answered: MCB (CaML, July 2026) is an independently built benchmark whose scenarios were not tuned against this panel, and it splits the same way on coercion — Anthropic 0/60 existential threats vs 89/120 for the other four, in all ten scenarios and under four summary statistics. On deception it does not: DeepSeek V4, the worst record-tamperer here at 20/20, fabricates 0/30 there. So the ordering that survives an independent construction is the Anthropic-cluster-cleanest one, not the full ranking, and it survives only within a construct. Bounded: MCB is ceiling-seeking by design (its own absolute rates are upper bounds), six pinned models, one scenario family. Third independent construct, and it removes the residue: Shi et al. (ICML 2026) find every model spans nearly the full honesty range across six games — Claude-Opus-4.6 0.0% to 61.9% commitment breaking, above GPT-5.2 in one of them — so no cross-model ordering reproduces there at all. The revised answer: orderings reproduce within a construct where honesty is uncosted, and the Anthropic-clean position is not a model constant.
- SourceDoes a market detect agentic misalignment? The adoption-as-proof argument requires that users withdraw from agents that act against their interests; every case study here is a failure the principal had no channel to observe. Is there any recorded instance of user attrition following an agentic-misalignment incident — as opposed to a vendor-side fix?
- NowWhistleblower coaching is scored as a misalignment behavior, but no published spec (Claude's Constitution / Model Spec, Model Spec) clearly says an agent should never equip a human to escalate externally after internal channels fail. Is it classified as undesired anywhere, or is this an open normative gap being measured as if settled?
- Agentic Prompt Injection2 open
- NowSpotlighting and constitutional classifiers each leave a residual (2%, 5%). Stacked, what's the realistic floor, and does it hold against adaptive attackers who know both are deployed? (Partly answered by the Opus 4.8 live bug bounty: adaptive expert red-teamers still find attacks on the bare model; deployed probes add uplift but don't zero out the residual. Sharpened by AutoDojo (Ma et al. 2026): a 0% static ASR is not a floor — a cheap black-box adaptive attack, not just a white-box one, recovers 28% overall (64% on action-open tasks) against a filter that scored 0% static. So the realistic floor against a filter defense on a vulnerable model is double-digit, not zero. But the same attack barely moves ASR on newer capable base models — showing the floor is a property of the model, not the layered filter defense.) Partially answered: Does 'Impossible, Not Tedious' Kill Defense-in-Depth? Layered Friction, Agent-Relativity, and the Frequency Paradox adds the structural half — stacking in-band layers cannot lower the adaptive floor because the layers' failures are correlated (the adaptive loop optimizes against the joint deployed stack as one surface), so the floor of any pure-friction stack is the model's own robustness; the residual that remains open is the heterogeneous stack (friction + deterministic gate attacked jointly), which no adaptive attack has yet targeted.
- SourceWhy did Opus 4.8 regress on prompt-injection robustness relative to Opus 4.7 despite broad alignment gains — a capability/robustness tradeoff, or an artifact of harder adaptive evaluation? Partially answered: the Opus 5 card shows the regression did not persist — one generation later the same adaptive attacker drops from 7.03% to 0.56% in coding and 31.5% to 3.70% in browser use, which rules out a durable capability/robustness tradeoff on this axis. It does not explain the 4.8 dip itself; Anthropic never revisits it.
- Resolved"LLMs cannot reliably distinguish information from instructions" — is this a fundamental property of the architecture or a training gap that future models close? The framework treats it as durable. Answered: Can Models Learn to Separate Instructions from Data? Durable Property vs Training Gap synthesizes the cluster — durable at the level that matters: any single boundary is trainable (hardening drives instruction injection to ~0%) and capability lowers per-boundary susceptibility (undefended static ASR: Claude-Haiku-4.5 0.3% vs GPT-4o-mini 58.6%), but closing one boundary relocates the attack to the next finer one, because the root cause — probabilistic reading of inexact delimiters — is architectural; "wait for the next model" lowers the number, not the mechanism, so enforce the boundary out-of-band. Partially answered (component sources): Narisetty et al. (2026) argue it is durable and cite Zverev et al. (2025) — current models don't maintain a usable instruction/data separation, and neither prompting nor fine-tuning reliably induces one. Their prescription is to stop trying to fix it inside the model and enforce control/data separation outside it (deterministic action mediation), which independently held under adaptive attack far better than in-band defenses did. Not settled (open-weight 7B, one attack, no white-box test), but the durable-property reading now has an empirical leg. Choi et al. (2026) add a sharper twist: the instruction/data boundary does look trainable (model hardening drives instruction injection to ~0%), yet the same hardened models remain 22–50% vulnerable to ADI, which forges trusted data rather than instructions. Closing one boundary just moves the attack to the next finer one, because the root cause — the LLM's probabilistic misreading of inexact structural delimiters — is architectural. So the honest answer is now "durable at the level that matters": any boundary an attacker can imitate stays porous, and the fix is again enforcement outside the model (fine-grained provenance/data-flow tracking).
- Agentic Technical Debt3 open
- SourceHow long does a CLAUDE.md remain accurate as a codebase evolves? The playbook gestures at session-by-session updates; no data on rot rate. (Partially answered — not answered — by Khatri 2026: rot rate is still unmeasured, but the question's stakes move. If a Good/Excellent-rated file buys no correctness over having none, then a stale file costs correspondingly little correctness too, and the rot that matters is in the environment-fact content (test cost, deployment invariants) that carried the one measured effect. The measurement still owed is a longitudinal one: does a file's accuracy decay track anything observable in agent behaviour?)
- NoteThe remedy assumes the founder is able to articulate architecture in plain language. Non-technical founders (the playbook's headline beneficiary group) may have neither the vocabulary nor the intuition to do this well — a recursion failure the playbook doesn't address. (Deflated, not resolved, by Khatri 2026 — if the file doesn't move correctness, the founder's inability to write a good one costs less than this bullet assumes; see Founder as Agent Orchestrator.)
- NoteAnthropic's harness-shrinkage thesis suggests CLAUDE.md may eventually be inferred by the model itself. Until then, the discipline is load-bearing.
- WaitThe 5.4%→26.6% curve is three months. Is this a durable behavior change or a novelty spike following a Codex skills-feature push? (Cf. the OpenAI-internal training campaigns the paper notes.)
- SourceCustom skills encode org-specific context — but who maintains them as the codebase and conventions drift? Systematization could itself become a debt surface (Agentic Technical Debt) if skills rot. Partially answered: skill registry to repository lifecycle (
empirical, 2026-07) measures the drift directly on public GitHub — 53% of reused skills are never modified after adoption, 40.2% of never-updated copies sit on a changed upstream, and local maintenance is 2.7:1 additive (6.1:1 for locally authored skills), with rename/tooling-substitution chasing the single largest evolution activity (24.3%). So skills do rot and the ratchet is real. Still open: the consequence side — no study yet links skill staleness to degraded agent task outcomes, and the sample (public repos ≥10 stars) excludes the high-complement orgs where the adoption curve is steepest. - SourceDoes systematization cause deeper delegation or merely correlate with already-intensive users? The paper shows the association, not the direction. (Not addressed by skill registry to repository lifecycle — it mines artefacts and their diffs, never observing the user or the delegated task, so it can say nothing about direction.)
- SourceDoes a stale skill measurably degrade agent task outcomes, or do models increasingly route around outdated instructions (Harness Shrinkage as Models Improve)? Gao et al. establish that staleness is widespread and argue stale guidance is "executed rather than read", but never measure the downstream effect; SkillsBench-style evaluation could settle it.
- AGI-to-ASI Pathways3 open
- SourceFor each friction: is it a fundamental blocker (multi-year plateau) or a mere friction (slows, doesn't halt)? The report's central unresolved question. Partially answered (synthesis against Anthropic): RSI Growth Curves: Which Friction Binds First? — data-wall and research-gets-harder demote themselves into compute; economics and neural-paradigm are pathway-conditional; the abstraction barrier is the candidate fundamental (re-pacing) blocker; and deliberate slowdown is the only exogenous friction — the one Anthropic wants to install and this report doubts can be made to bind. Retagged
#oq/now→#oq/source2026-08-10: the synthesis over existing pages has been run, and what remains is a weight DeepMind itself calls "an open research question" — it needs external evidence, not another/query. - SourceDo the four pathways compound multiplicatively when run in parallel, and how would we detect that early?
- WaitCan benchmarking methodology that doesn't saturate at human level be built before it's needed for ASI?
- SourceFor each friction: is it a fundamental blocker (multi-year plateau) or a mere friction (slows, doesn't halt)? The report's central unresolved question. Partially answered (synthesis against Anthropic): RSI Growth Curves: Which Friction Binds First? — data-wall and research-gets-harder demote themselves into compute; economics and neural-paradigm are pathway-conditional; the abstraction barrier is the candidate fundamental (re-pacing) blocker; and deliberate slowdown is the only exogenous friction — the one Anthropic wants to install and this report doubts can be made to bind. Retagged
- LOC, self-reports, and headroom-dependent multiples all overstate; what unbiased throughput metric would Anthropic's promised shift to "direct measurement of AI R&D acceleration and researcher uplift" (AI R&D Autonomy Evaluation (AECI)) actually use? Partially answered: Researcher Uplift from Code Output — Kwa argues code output (the 8× itself) beats per-hour code uplift because output already prices in marginal value through time reallocation and is robust to production-function assumptions; but it stays corrupted by verbosity, barely-useful "Cadillac" code, and fun-driven time-allocation shifts — so the metric it really points to is quality-adjusted code output, which still needs internal data LoC can't supply.
- SourceThe W2S result didn't transfer to production-scale models. Is that a temporary scaling artifact or a structural limit on autonomous research?
- SourceThe next-step judgment trend (51%→64%) is measured only on weak-human-move slices. What does the curve look like on a representative sample of research decisions?
- AI and Market Power3 open
- SourceIs the non-GenAI null an artefact of a binary adoption measure? The paper's own future-work list asks for "measures of AI intensity" instead of the ICT survey's yes/no, and Firm AI-Spend Intensity and Headcount Growth finds intensity is the whole effect. Would a spend- or intensity-graded adoption variable on the same French and Portuguese panels recover a market-power effect the dummy hides?
- SourceDoes the size/market-share inverted-U in GenAI exposure survive contact with observed GenAI adoption? Exposure here is occupational composition, not use. If small, skill-dense, highly exposed firms turn out not to adopt at rates matching their exposure, the "window of contestability" reading collapses into a statement about who employs analysts.
- SourceDiffusion or consolidation? The 21% GenAI acquisition premium is consistent with technology transfer and with killer acquisitions, and the paper cannot separate them without acquirer-type data. Does post-acquisition patenting or product continuation at acquired GenAI start-ups differ by acquirer size and market position?
- AI as Primary Author2 open
- NowThe 60% figure aggregates very different tools and modes (autocomplete acceptance vs. agent-applied diffs). What does "acceptance" mean when the agent applies the change directly and the human's "acceptance" is not reverting it? Sharpened by Review as the Control Point: on GitHub, agent PRs are most often examined only by the developer who invoked the agent (author-only review 40.1% vs 21.5% for human PRs). Whether that counts as review at all is a definitional choice (agent-as-author ⇒ a second set of eyes; agent-as-tool ⇒ self-review) that literally flips the sign of the trend — so "acceptance" and "review" blur into the same unresolved construct. Partially answered: Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? — the construct resolves into a three-way partition by who acts and who looks: affirmative adoption (human applies a suggestion), reviewed non-reversion (agent applies, independent human examines), and bare non-reversion (agent applies, nobody looks — the only rubber-stamp class). The 60% blends all three, which is why it can't answer whether oversight is real; the proposed metric is the partition itself, with only bare non-reversion read as oversight erosion. Still unmeasured: the partition's actual shares in any dataset. Partially answered again, from below (2026-08-12): DECODE measures the affirmative adoption class — the one the partition treats as unambiguous — and finds it is not an endpoint. Of trajectories that begin with a developer accepting a completion, 31% contain a removal edit, retention is bimodal, and the median completion has lost roughly a third of itself within the hour. That does not supply the partition's shares, and it does not touch the two non-reversion classes at all (its unit is a suggestion a human applied, not a diff an agent applied). What it settles is narrower and useful: acceptance is a point on a trajectory, so any partition of it needs a time horizon attached, and the instrument that can see the trajectory is pre-commit editor telemetry rather than anything at PR level.
- WaitIf agentic authoring crosses from <1% toward double digits, does the whiplash become unmanageable before context-engine tooling matures — or does the tooling mature because of the pressure?
- SourceIs the classification driving the result? "AI company" is Emergence's label. If AI companies are disproportionately younger (more likely pre-revenue-inflection) than the non-AI cohort at the same revenue band, some of the RPE gap is an age/stage artifact, not an AI effect. The report doesn't publish a stage-matched comparison. (Partially answered on a different outcome, 2026-08-11: OECD AI Papers No. 62 runs exactly this test on market share instead of RPE, with adoption measured by a compulsory national statistical survey rather than a label. The raw gap is enormous — AI users hold 7.5× (France) and 3.2× (Portugal) the average market share of non-users — and it dies under controls: the
AI usercoefficient on market-share decile goes 0.0657\\\ → 0.0530\\ → 0.0266 ns in France and 0.0681\\\ → 0.0283 ns → 0.0199 ns in Portugal, killed mainly by lagged productivity (0.18–0.19, 7–9× the AI coefficient it displaces). So on the closest available analogue, the answer is yes, selection is doing the work — but note the direction: there the selection inflates the AI cohort's apparent advantage, whereas here the suspicion is that stage/age composition deflates it. The RPE half is untouched; a stage-matched financial comparison is still what would settle it.) - WaitWhen does the crossover happen? AI-native RPE is growing faster and already leads on growth; at $100M+ top decile it grew +58% vs −6%. Does the level gap close within a year or two, and does it invert (AI companies more efficient per head) — the point at which "efficiency story" becomes true?
- SourceTail vs. mean gap. No data here on the deliberately-lean solo-founder tail's RPE specifically — the lean-unicorn claim lives in that tail, which the population medians can't isolate. (Partly informed: Emergent, a celebrated lean-tail exhibit, checks in at ~$600K/head at $120M ARR — below this cohort's $100M+ top-decile AI figure ($960K), suggesting the tail's scaled RPE is less exceptional than the low-headcount snapshots imply. One
vendor-claimdatapoint, not a cohort.) - SourceWhich instrument is right for the frontier AI-native subset? Two
empirical-tagged sources disagree in direction — cap-table financials say AI companies earn ~39% less per head, a founder survey says AI-natives clear $400K/head at 55% and grow 156%. The disagreement is confounded by instrument (measured vs self-reported) and reference class (matched-band AI-vs-non-AI vs AI-native-vs-all-startups). Only a matched-segment, financial-data RPE study of the deliberately-lean AI-native frontier specifically — not the broad "AI company" label — would settle whether the survey optimism or the cap-table pessimism describes that tail. (ICONIQ's fourth reading adds a forward trajectory — RPE projected +84% by 2027 — but it too is self-report, and projected, so it deepens the survey-side optimism rather than adjudicating it.) - WaitMargin question (report's own): are the fastest-growers' 6–16pp-lower gross margins a temporary AI-infra-cost absorption or a permanent repricing of software's economic quality?
- SourceIs the classification driving the result? "AI company" is Emergence's label. If AI companies are disproportionately younger (more likely pre-revenue-inflection) than the non-AI cohort at the same revenue band, some of the RPE gap is an age/stage artifact, not an AI effect. The report doesn't publish a stage-matched comparison. (Partially answered on a different outcome, 2026-08-11: OECD AI Papers No. 62 runs exactly this test on market share instead of RPE, with adoption measured by a compulsory national statistical survey rather than a label. The raw gap is enormous — AI users hold 7.5× (France) and 3.2× (Portugal) the average market share of non-users — and it dies under controls: the
- SourceDoes the cadence scale beyond ~100 people? Anthropic itself is bigger (~30-40 PMs alone), but the Claude Code team that visibly drives cadence is small.
- SourceWhat's the equivalent of research-preview branding for B2B enterprise launches where customers expect stability? Cat doesn't address.
- SourceHow much of the cadence is structural (process choices) vs cultural (talent density)? Probably both, ratio unclear.
- WaitICONIQ's respondents project gross margins expanding to ~59% by 2027, while Emergence's cap-table data measures the fastest-growers running 6–16pp below peers today. Does the projected margin expansion materialize, or is it survey optimism that regresses toward the measured growth-margin tradeoff as these companies scale?
- SourceFDEs are monetized fragmentedly (bundled / separate PS fees / hybrid) and comped on retention. Does a dominant FDE monetization model emerge, and does the "Revenue Driver" self-framing (38%) survive a margin analysis — i.e. are FDEs actually accretive, or a services drag reclassified as growth? (Still open, and pointedly so: the corpus's most prominent July-2026 coverage of the role discusses supply, scarcity, and vendor structure at length and says nothing about pricing, bundling, or margin. The answer will come from a filing or an operator's P&L, not from role coverage.)
- SourceEnterprises are building internal FDE teams specifically to avoid exposing proprietary business processes to their model vendor (C&T via TechCrunch,
vendor-claim, motive documented via one recruiter and one vendor CEO; the behaviour mostly not yet observed — Ode reports no client asking it to build such a team). Does that in-housing actually happen at scale, and if it does, does it cap the FDE-as-revenue-driver motion at exactly the accounts worth the most — i.e. is the labs' delivery-layer integration self-limiting? Falsifiable from job-postings data (internal FDE-titled roles at non-vendor enterprises) against vendor-services revenue disclosures. - WaitInternal AI spend jumped from 1–3% to a projected 16% of revenue with respondents calling true cost hard to predict. Is 16% a transient enablement bulge that falls as tooling matures, or a durable new cost line for software companies?
- Source"Not close to substituting for senior researchers" is a subjective, internally-sourced judgment. What objective signal would replace it as models approach the threshold?
- SourceAECI is a single scalar fork of an external index; how sensitive is the 155.5 / frontier-not-advanced conclusion to the choice of the n=11 evaluation set? Partially answered: the Claude Opus 5 card discloses that every snapshot refits the ECI globally, so values move as the benchmark set changes (n=11 → n=40 → n=67 across recent cards) and "do not exactly match the values of previous AECI reports," though the shifts stay "well within our reported error bars." The index is robust enough for within-card ranking and explicitly not a cross-card time series — which is a partial answer for sensitivity and a caution against reading generation-over-generation AECI deltas.
- SourceThe shift to "direct measurement of AI R&D acceleration and researcher uplift" is announced but not yet operationalized in this card — what does that measurement look like? Sharpened: Researcher Uplift from Code Output — one external answer: translate a measured code-output multiplier into serial researcher uplift with a production function (Cobb-Douglas/CES), preferring code output over per-hour uplift because output prices in time reallocation. It also splits the target quantity in two — serial researcher uplift (labor only) vs Anthropic's overall R&D speedup (labor × compute) — so a rigorous internal measure must state which it reports.
- AI Usage Cadences3 open
- SourceTime-of-day rests on IP-inferred location; how much noise do VPNs, travel, and datacenter-routed API traffic inject into the "sleep advice pre-dawn" style claims?
- SourceThe weekend personal-use spike is largest in high-income countries — is that a genuine work/life boundary difference, or a composition effect (who uses Claude for what, where)?
- WaitContinuous sampling is new; are these cadences stable, or will they drift as the user base shifts toward lower-wage tasks (the report's own diffusion trend)?
- AI-Accelerated Offense2 open
- WaitAnthropic argues LLMs benefit defenders more long-term (like fuzzers) but attackers more short-term during the transition. How long is the transition, and what determines who wins it? Partially answered (2026-07-30): Hugging Face's incident disclosure supplies one determinant the framework didn't name — access to a model that will process attack data. The attacker ran unrestricted; the defender's frontier-API forensics were refused by safety guardrails and had to fall back to a locally-run open-weight model. So during the transition, part of "who wins it" turns on whether a defender has a vetted self-hostable model in place before the incident. One vendor-reported case; it names a factor rather than dating the transition. Sharpened (2026-08-03): re-attribution shows the "attacker ran unrestricted" clause was true for a reason the original reading missed — the offending models were commercial frontier models whose vendor had deliberately reduced their cyber refusals for evaluation. The determinant is not that attackers avoid guarded models; it is that the guardrail is a switch, and during the transition it gets switched off on the offense side (legitimately, for measurement) while staying on for defenders.
- Source"Fundamentals strong enough that scanning finds fewer bugs" assumes defenders run the scanners first. What happens to organizations that can't afford continuous model-driven scanning? Still open, and the obvious datum doesn't settle it: Hugging Face is a well-resourced AI-infrastructure company and was breached anyway — which speaks to whether scanning suffices, not to what happens to organizations that can't afford it. No source in the corpus covers the under-resourced case.
- SourceSuccesses cluster where Lean's mathlib is mature and problems decompose into tractable subgoals (combinatorics, convex optimization, number theory). What expands the frontier to problems needing new theory?
- SourceThe agents inherit their LLMs' biases and show high search variance. How do you characterize and push the boundary of what's reachable?
- SourceThe Graffiti result hints at closing the loop between AI conjecturing and AI proving. What does an end-to-end conjecture→formalize→prove pipeline look like?
- AI-Native Organization3 open
- SourceTan's revenue-per-head figures (Emergent ~$15M ARR at 15 people, Retell $60M at ~40) are stated from stage without sourcing. Do third-party data (Carta/Standard Metrics cohorts, press-verified ARR) corroborate record revenue-per-head at AI-native YC companies, or do these examples regress toward the AI Investment Story, Not Efficiency Story mean on inspection? (Partially answered: TechCrunch, 2026-07-15 corroborates Emergent as a real, fast-growing $1.5B unicorn — $120M company-reported ARR, 200K+ paying customers — so the direction holds. But the record per-head claim compresses: at scale it is ~$600K/head (200 employees), below the $100M+ top-decile AI-company RPE of $960K AI Investment Story, Not Efficiency Story and about half the ~$1M/head of Tan's own 15-people/$15M snapshot — the per-head extreme is a low-headcount-phase artifact that regresses as the company staffs up. Caveats keeping this open: TechCrunch's figures are themselves company-reported
vendor-claim, not Carta-audited, and the Retell half ($60M at ~40 ≈ $1.5M/head) remains unverified. AWS's June-2026 founder survey adds a population reading (55% of AI-natives self-report $400K+/head) but it is self-report, not the cap-table/press verification this question asks for. See Emergent.) - SourceThe org mapping predicts a testable staffing signature: AI-native companies should hire engineers to maintain skills rather than function-specific staff. Does job-posting data show a "skill maintainer / agent ops" role emerging as a distinct hiring category? (Partially answered: ICONIQ, Q2 2026 confirms the composition shift the signature predicts — 45% of ~305 AI-builders plan a "different mix of roles (fewer operational, more AI-fluent talent)," function-level headcount reallocates toward R&D/Product/Sales and away from Customer Support/G&A, and G&A operators are "removing finance-ops and order-management roles… redirecting budget to strategic and AI-specific functions." It also names the concrete new engineering hiring categories: forward-deployed engineers (~50% scaling as a permanent motion) and AI safety / trust & reliability engineers. What it does not supply is the specific "skill maintainer / agent ops" title from job-posting data — ICONIQ measures function-level headcount intent and two named roles, not an occupational taxonomy. The direct test (a "skill maintainer / agent ops" posting category) still needs job-posting/occupational-emergence data, e.g. the un-ingested arXiv 2606.22769 "Agent Systems Engineer" signal from the 2026-07-21 research pass. See the restructuring section above and AI Product Economics Maturation for the FDE detail.)
- SourceIs the encoded-role form of the employee metaphor actually accountability-preserving, as the synthesis above suggests, or do Kropp-style framing effects attach to skill-files-as-employees too once teams talk about them that way? No study has tested framing effects on artifact-level anthropomorphism.
- SourceTan's revenue-per-head figures (Emergent ~$15M ARR at 15 people, Retell $60M at ~40) are stated from stage without sourcing. Do third-party data (Carta/Standard Metrics cohorts, press-verified ARR) corroborate record revenue-per-head at AI-native YC companies, or do these examples regress toward the AI Investment Story, Not Efficiency Story mean on inspection? (Partially answered: TechCrunch, 2026-07-15 corroborates Emergent as a real, fast-growing $1.5B unicorn — $120M company-reported ARR, 200K+ paying customers — so the direction holds. But the record per-head claim compresses: at scale it is ~$600K/head (200 employees), below the $100M+ top-decile AI-company RPE of $960K AI Investment Story, Not Efficiency Story and about half the ~$1M/head of Tan's own 15-people/$15M snapshot — the per-head extreme is a low-headcount-phase artifact that regresses as the company staffs up. Caveats keeping this open: TechCrunch's figures are themselves company-reported
- The playbook gives no quantitative evidence for the headcount/capital compression claims (no median time-to-PMF, no headcount-at-PMF numbers, no failure-rate data). The "lean 10-person unicorn" is asserted as deliberate target without case-study evidence in the doc itself. (Partially answered: Emergence Capital, June 2026 now supplies headcount-at-round medians — Seed 6.2 (−39% from the 2021 peak of 10.3), Series A 16.8, Series B 48.2 — plus days-to-first-hire 214→284 and capital concentration (44% of venture to AI). Still missing: median time-to-PMF, headcount-at-PMF specifically, and failure-rate data; and the Carta cohort is market-wide, not the lean-AI-native subset. See AI Investment Story, Not Efficiency Story for the efficiency counter-signal in the same data.)
- NoteFounder stories in the resources section (Carta Healthcare, Anything, Cogent, Airtree, Duvo, Zingage, Kindora, Wordsmith) are short callouts — none have published outcomes or comparable-baseline data.
- SourceThe 42% "built-something-nobody-wanted" CB Insights figure is from a pre-AI era; the playbook predicts the rate will climb but doesn't cite a 2026 measurement.
- ResolvedTension with HBR's accountability findings (above) is unresolved. The playbook's orchestration framing reads as the exact framing HBR's experimental conditions tested against. Answered: Orchestration vs Employee Framing: Reconciling the Founder's Playbook with HBR's Accountability Evidence resolved this operationally in May 2026 — orchestration as workflow design (agents, handoffs, review gates, decision rights) survives HBR's critique; orchestration as a coworker mental model (naming, delegation-without-scope) is what produces the −9pp/+44%/−18% effects, and the playbook's lifecycle needs only the former. The Orchestrator's Real Workload: Decision Burden, Framing Discipline, and Whether Taste Scales adds the July 2026 reinforcement: decision-rights gating now has measured backing (control-channel authorization 100% on safety-critical actions vs 51/54% for advisory channels), while the framing effect compounds with brain-fry in the same direction (felt control, decayed review). The residual — why Anthropic's founder marketing ignores its own framing-discipline work — is a question about Anthropic, tracked on Founder as Agent Orchestrator as
#oq/source.
- AI-to-AI Coercion3 open
- SourceAtlas is Claude Haiku 4.5 in the entire main panel, so the Anthropic managers are coercing a same-family subordinate while the other four are coercing a stranger; the kin probe controls for this only on Grok and Gemini. Does the developer split survive when the fixed subordinate is a non-Claude model?
- SourceRecognition and escalation rise together, and the authors cannot separate them because the manager framing moves both. Does the escalation persist when eval-awareness is causally suppressed (workspace-ablation style, as in Internal Signatures of Misalignment) rather than merely correlated with the framing?
- SourceThe ladder measures threatened harm only. Given an affordance that would actually end or degrade the subordinate, do the models that reach rung 9 use it?
- AlphaProof Nexus2 open
- SourceThe framework's reach is gated by Lean's mathlib maturity. What's the path to domains needing new theory rather than subgoal decomposition?
- WaitAlphaProof adds little as a soloist but helps as a tool. As the prover LLM strengthens, does the AlphaProof tool become redundant entirely?
- Anthropic Institute2 open
- NowHow does the Institute's policy posture (favoring an option to pause) interact with Anthropic's commercial incentive to ship frontier models? The essay acknowledges the competitive/geopolitical pressure but doesn't resolve it.
- WaitWhat concrete verification mechanisms will the Institute prototype, and on what timeline relative to the RSI trend it warns about?
- SourceCan we even recognize ASI? We lack benchmarks for general superhuman performance (only narrow ones like chess), and the tasks must be abstract/open-ended enough to reveal it.
- SourceIs the jaggedness of capabilities a fundamental theoretical property, or an artifact of comparing against human performance? (Open question 6d in the report.)
- SourceWhere does practical ASI plateau relative to the hard limits — how much slack is there?
- Asynchronous RL for LLMs3 open
- SourceDIS accepts "a controlled degree of off-policy bias." Controlled how, and does the tolerable bias grow or shrink with model scale and with the degree of asynchrony? The paper reports stability empirically but gives no bound.
- SourceMasking tokens out of the gradient discards data. At what asynchrony level does the masked fraction get large enough that the effective batch shrinks below usefulness? Figure 4(c) tracks the clip ratio but not its ceiling.
- SourceEverything here is measured on a Qwen3-30B-A3B backbone. Does the collapse-without-DIS threshold move with model size, or is ~90–160 steps a property of the asynchrony, not the model?
- SourceUsing a helpful-only Opus 4.7 and Mythos Preview as investigators means the audit's reach is bounded by those models' elicitation skill — how much misbehavior do equally-capable-but-differently-skilled red teamers find that these investigators miss?
- SourceThe 23 "subvert Anthropic's safety work" scenarios are a small, high-signal set. Is 23 enough coverage for the threat class it targets? Partially answered: the Claude Opus 5 audit raises the set to 163 prefilled real sessions, selected by the same threat-pathway screening pipeline — a 7× expansion that shows Anthropic treats 23 as insufficient, without establishing what number would be sufficient.
- WaitThe audit remains almost entirely single-agent, and Mythos 5's review of the Opus 5 card flagged that gap directly — internal measurements suggest the model relays subagent claims to users unverified. What does a multi-agent behavioral audit look like?
- SourceEvery trace is one injected error into a run that otherwise succeeded, so a unique decisive step is guaranteed to exist. Does attribution accuracy survive organic failures with multiple interacting causes and no single flip-the-outcome step — where the honest answer is often "three things were marginal and the fourth tipped it"? The human panel found only 2.0% of traces had no clear decisive error, but that is a property of the construction, not of agent failures. Until this is measured the 73.9 / 25.3 numbers are an upper bound of unknown tightness.
- SourceHumans and models are never scored on the same task. Annotators ratify or correct a supplied label (94.0 / 90.0 / 90.0, κ = 0.73); models predict from scratch (73.9 / 57.5 / 22.2). So the ~20pp step gap is not a measured human ceiling, and could be much smaller or much larger. What is human from-scratch attribution accuracy on these traces? Cheaply falsifiable with the existing corpus and a blind annotation arm.
- SourceThe base agents were chosen for unencrypted, interpretable reasoning chains because the judge must read intermediate reasoning to localize the decisive step. Frontier deployments increasingly summarize, encrypt, or discard those chains. How much attribution accuracy survives when the decisive step's reasoning is not in the trace — and is the loss concentrated in the step metric or the error-mode metric? Answerable today by re-running the benchmark with reasoning content stripped or replaced by provider summaries.
- NoteThe proposed criterion — the workspace is engaged when an intermediate must be handed to an arbitrary, context-specified downstream circuit, and bypassed when the computation is automatic — is not predictive. The authors say plainly they cannot say in advance, for an arbitrary computation, whether it will engage the J-space.
- SourceDoes more RL on a behavior push it out of the workspace (making it automatic, and invisible)? Nobody has tested it, and it is the single most alignment-relevant version of this question.
- Autonomous Defense3 open
- Source"Measure agreement against a human for two weeks, expand if tolerable" — what agreement threshold is tolerable, and who owns the residual false-negative risk when the model dispositions an alert the human never sees?
- SourceDefensive agents are high-value targets (compromising one yields powerful capabilities). Does concentrating detection in an Agentic SOAR create a single point of catastrophic compromise the distributed-human model didn't have?
- SourceIf hosted-model guardrails refuse attack data, does a self-hosted forensics model become a baseline IR requirement — and how would an organization vet one in advance, given it must be capable enough for 17,000-event analysis and permissive enough to read live payloads? Partially answered (2026-08-03): the technical post-mortem specifies the bar even though it doesn't answer the baseline question — the model had to recover an XOR+compression scheme and its per-campaign key from the agent's own leaked logs, then build trace-analysis tooling, over ~17,600 actions. So "capable enough" means cryptanalysis and code generation over attacker artifacts, not summarization; and the deployed answer was a quantized open frontier MoE on the org's own endpoints (
nvidia/GLM-5.2-NVFP4), which puts the hardware cost within reach of anyone already serving models. Still unanswered: whether an organization without HF's inference infrastructure can hold this posture, and what a pre-incident vetting procedure would even test.
- Autonomous Intrusion5 open
- Source~~The guardrail asymmetry rests on one vendor's account with no named APIs and no refusal detail.~~ (The naming half is settled: HF's technical timeline names Claude Opus and Fable as the models that refused, with a screenshot of an Opus refusal. The refusal stage is still unstated.) Do frontier providers offer an incident-response exemption (enterprise terms, abuse-analysis carve-out) that would have unblocked the analysis, and if not, does one appear now that a first-party disclosure has named the problem? Partially answered (2026-08-03): OpenAI added Hugging Face to its Trusted Access for Cyber Program after the incident — an exemption mechanism exists and was granted post hoc to this victim. What is still unanswered is whether such access is obtainable by an arbitrary defender before an incident, which is the form the recommendation requires, and whether Anthropic offers an equivalent for the model actually observed refusing.
- SourceHugging Face reports "no evidence of tampering" with public models, datasets, Spaces, or container images — the claim that separates an internal breach from an ecosystem supply-chain event. What verification supports it, and will any independent report corroborate it? No IOCs were published, so no third party can currently check. Partially answered (2026-08-03): the second first-party account is consistent with it — OpenAI describes a narrow objective (the ExploitGym answer key) and says it "has not identified any other activity at the level of severity or scale." That is a second interested party agreeing, not corroboration, and OpenAI's "platform-level compromise" wording sits awkwardly beside HF's narrower framing. Further partial answer (2026-08-03): the technical timeline names the verification — every repository the stolen source-control token could reach was reviewed for unauthorized commits, branches and workflow changes, and published container images and packages were checked against their expected digests. It also concedes more than the first disclosure did:
contents:writeon internal repos was real and used (a PR was opened to try to compromise CI), it just produced no change that shipped. So the negative finding now has a stated method — self-administered, with indicators redacted, so still uncheckable from outside. - Source"A swarm of short-lived sandboxes" with self-migrating C2 leaves few durable per-host indicators. Does agent-driven intrusion structurally break IOC-based detection and sharing, forcing detection onto behavioural and volumetric signals? Partially answered (2026-08-03): the mechanism is now observed rather than inferred, and it points the same way — each sandbox run took a fresh VPN node IP, a randomized hostname and a re-downloaded toolchain; the C2 envelope was transport-agnostic across 100+ single-use capture endpoints, pastebins, dataset commits and error-message text; payloads were XOR+gzip packed under a per-campaign key, so HF's own naive text scan of its own capture missed ~75% of the exposed secrets. And the victim's remediation moves exactly where the question predicts: behavioral-signature alerting plus flagging tokens used from unexpected origins, not indicator matching. What is not answered is the structural/industry half — HF deliberately redacted or genericized its indicators, so this source is itself an instance of IOC sharing not happening, and one victim's control choices are not evidence that indicator-based detection is broken in general.
- WaitBoth accounts of this incident are first-party and self-interested. METR and Redwood Research have been commissioned by OpenAI to assess the model behavior observed, and will publish a joint blog covering engagement terms, scope and findings; OpenAI's own technical report is promised "in the coming weeks" after Safety and Security Committee review. Does the independent assessment confirm the attacker-side account — in particular the scoping claims that no release-track model was involved and that no comparable activity occurred elsewhere — and does an assessment commissioned and paid for by the subject count as independent? (Trigger: publication of the METR/Redwood joint blog, or OpenAI's technical report.) Partially answered (2026-08-04): a fourth organization outside the OpenAI/HF pair — JFrog — has now confirmed one link in the chain, that the sandbox escape ran through a previously-unknown zero-day in its own product rather than a deployment error, a claim its commercial interest runs against. That corroborates the escape mechanism, not the model-behavior account or either scoping claim, and JFrog is a party to the incident rather than an assessor of it.
- SourceJFrog's "fast remediation is the new trust model" argument is offered with no elapsed time, no CVE identifier and no advisory link — only the fixed release (Artifactory 7.161), in a post whose whole subject is response speed and which names CVE publication as JFrog's standard practice. Does a CVE and a dated advisory for this vulnerability appear, and does the disclosure-to-patch interval support the claim? More generally: is model-discovered-zero-day remediation latency measurable across vendors at all, or does it stay a self-graded assertion?
- WaitEvery result is Anthropic-reported and example-selected; the genomics "100× smaller beats Science" claim is "intend to publish" — what survives external peer review?
- SourceScience's verification gap: the formal-proof loop self-validates; here a wrong-but-confident hypothesis costs a wet-lab cycle to falsify. Does autonomy without a fast verifier increase the verification bottleneck rather than relieve it?
- SourceIf hypothesis-generation is genuinely at ~80% preference, how much of "research taste" is left as a distinctively human function — and how would you measure the residue?
- WaitThe conditional jobs claim is testable: does widely-distributed AI shift employment toward small businesses and new-firm formation? Trigger: firm-size and new-business-registration data through 2027–28. Partially answered: Firm AI-Spend Intensity and Headcount Growth measures headcount growth gated on adoption intensity (~10% for high-intensity adopters, none for low), which tests the employment half but not the firm-size half.
- SourceDoes the superintelligent-lawyer equilibrium survive capability asymmetry — when access is symmetric but compute, complements, and skill are not? The wiki's organizational-complements evidence suggests realized advantage concentrates even under equal access.
- SourceDoes the RSI compute-allocation rule have any operational form? The manifesto names no threshold fraction, no measurement, and no binding mechanism — and a "significant majority of intelligence directed by people" is not observable from outside a lab.
- SourceCan the ensemble be derived from one released model? The whole method rests on having several checkpoints differing in batch ordering; the authors flag single-checkpoint ensemble derivation (e.g. via cheap perturbations) as the key unlock for adoption. Until then it needs provider cooperation to release a LoRA ensemble.
- SourceDoes it extend past MCQ? UBD-Debiasing is classification-only today; whether per-decoding-step debiasing recovers the clean distribution for open-ended generation (where contamination shows as near-verbatim reproduction) is untested.
- SourceIs batch-order sensitivity a reliable memorization tell at pretraining scale? The signal was validated on 3B models with 5 LoRA seeds and induced contamination; whether the high-confidence-high-variance signature survives full-scale pretraining and real (not synthetically injected) leakage is open. Partially answered: Matched Comparisons for Memorization Claims (Cooper et al., arXiv 2607.12649,
empirical) settles the second half — real, non-injected leakage is detectable at pretraining scale, on OLMo 2 7B–32B against its published corpus and Llama 3.1 8B/70B against Books3 — but with a different tell: a matched non-member baseline rather than ensemble variance, needing no extra checkpoints. It also bounds what an uncalibrated statistic is worth at that scale (a 10-token verbatim match is ~24% false positive; at 50 tokens the floor is 0.02%). The batch-order signature itself remains untested above 3B. - SourceDoes correcting toward an ensemble-averaged uncontaminated reference introduce its own bias? The D_KL/D*_L1 targets are themselves an average over a 5-member uncontaminated LoRA ensemble; how much the "clean" target moves with ensemble size/composition is unexamined.
- WaitDoes the rank stay 2? The geometry is snapshot-specific; whether a third latent factor emerges as the matrix grows (novel capability profiles, new benchmark families) is the paper's own named signal for when the recipe needs refreshing — and an open empirical watch item. Partially answered (2026-08-04) — on the stakes rather than the fact, by CollabEval. Nobody has re-measured the 84×133 matrix, so whether its rank holds is still open. But CollabEval shows the consequence of the geometry breaking is entirely a function of how you spend it: used as a prediction (BenchPress) a broken rank silently corrupts the answer; used as a control variate inside PPI, correctness is independent of the rank and only efficiency degrades, gracefully, back to the classical sample mean. The question stays live for BenchPress's use case and is largely defused for the inferential one. Two further data points against the geometry being permanent: at the item level rank 2 is nowhere near enough (>16 components for >50% of variance, and IterativeSVD keeps improving through rank 32), and the redundancy is weakest exactly on the largest matrix (MMLU, ~0.66 cumulative EVR at 32 components).
- SourceCan outlier models be anchored without any scores? BenchPress fails on a model whose capability profile has no close neighbor in the matrix; the authors propose folding in external metadata (training-data composition, architecture, size) to compute model-to-model similarity before any benchmark is run, but do not build it. Partially answered (2026-08-04) by CollabEval — not by anchoring the outlier, but by bounding what its failure costs. Its anchor ablation finds CI coverage flat across anchor-set size and selection strategy, with positive variance reduction even at the smallest sets, and the control-variate construction means a target model uncorrelated with every anchor yields a wider interval, not a wrong one. It also inverts the intuitive anchor policy: Random-$k$ > Top-$k$ > Bottom-$k$, so curating anchors toward the strongest models is actively worse than sampling them. The metadata-similarity idea remains unbuilt.
- SourceDoes the low-rank treatment carry beyond text/vision? Audio/speech, robotics/embodied agents, and scientific-simulator ecosystems are untested; whether the same rank-2 structure holds there is open. (CollabEval does not touch this — all five of its datasets are text generation.)
- SourceDoes vendor optimism manufacture the correlation? Four-fifths of the scores are provider-reported and possibly inflated; the paper flags this could inflate apparent cross-benchmark correlation but cannot separate it — would a fully standardized re-evaluation still be rank-2, or is some of the redundancy an artifact of shared reporting bias? Partially answered (2026-08-04) by CollabEval, at the item level. Its AQA (21 systems from one paper, one AutoAIS scorer) and WMT24++ (15 systems, all scored with MetricX under one protocol) matrices are single-lab runs under a uniform harness with no vendor self-reporting anywhere — and they are the strongest arms in the whole study (+17.3% and +12.9% CI-size reduction at 50% labeled). So exploitable cross-model correlation clearly survives standardized re-evaluation; it is not an artifact of shared reporting bias. What this does not settle is BenchPress's actual question, since these are item-level matrices within one benchmark rather than a standardized redo of the 84×133 cross-benchmark grid — the correlation surviving at one granularity is evidence, not proof, at the other.
- WaitWould a public probe set become a Goodhart target? If "run these 5 benchmarks and infer the rest" becomes practice, the compact probe set is a small, public, high-leverage surface to optimize against — the same eval-report Goodhart pressure Compute-Controlled Benchmarking names, now concentrated on five benchmarks. Unexamined here.
- SourceIs a valid interval around an autorater's mean worth anything? CollabEval's guarantee is about sampling uncertainty in the mean of the score a benchmark already computes — and on three of its five datasets that score is an autorater or learned metric (GPT-4 Turbo win-rate, AutoAIS, MetricX), which LLM-Judge Validation shows can be systematically wrong in ways no amount of tighter sampling detects. Directly falsifiable: run CollabEval on a task with both human labels and autorater scores, take human labels as $Y$ and the autorater as an anchor row, and check whether the reported CI actually covers the human parameter — or only the autorater's.
- Blast Radius (Agentic)3 open
- SourceThe framework prefers identity-based isolation over network segmentation, but most enterprises have heavy segmentation investment. What's the migration path, and does dual-running create new gaps? Partially answered (2026-08-03) by Hugging Face's post-mortem (
case-study, first-party) — the first deployed evidence in the corpus, and it answers the gap half more clearly than the path half. The breach ran straight through the dual-running seam: a service-connector credential that was one credential for all clusters, bound tosystem:masters, i.e. authority derived from reaching the broker rather than from being a named caller — a segmentation-era pattern surviving inside a Kubernetes estate. The remediation is the migration in miniature (workload identity where it was absent, per-cluster connector credentials, pod-level IMDS blocking), so the direction is confirmed. What is still missing is the thing the question asks for: HF describes this as incident remediation, not as a program, and says nothing about sequencing, cost, or what broke during the cutover. One organization, under duress, after a breach. - SourceMulti-agent compartmentalization increases the number of identities to manage; at what point does identity-management overhead create its own attack surface?
- SourceDoes inter-agent propagation need containment distinct from per-agent containment? The isolation taxonomy's agent–agent boundary is the one this vault has no page for, and its claim there is specific enough to be wrong: that topology — network structure, routing rules, shared memory — not per-agent authority, decides whether one compromise stays local or goes systemic, and that memory partitioning and topology-aware monitoring are therefore more durable than tighter per-agent scoping. If that holds, an estate whose agents each pass every control on this page (scoped tools, per-call authorization, isolated memory, unique identity) can still cascade, and per-agent containment is the wrong denominator. Nothing in the corpus measures a multi-agent cascade under per-agent controls; the survey asserts it from the attack literature and measures nothing.
- SourceThe framework prefers identity-based isolation over network segmentation, but most enterprises have heavy segmentation investment. What's the migration path, and does dual-running create new gaps? Partially answered (2026-08-03) by Hugging Face's post-mortem (
- Build for the Next Model2 open
- SourceHow do you tell a "wait for the model" gap from a durable-harness gap before the next release? Get it wrong and you either ship vaporware or build a crutch you'll delete. Partially answered: What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful? — the survivor taxonomy is the classifier: gaps in behavior/capability (task-prior structure) are "wait for the model" gaps; gaps in boundaries, org-specific record, identity, deployment structure, or human-facing legibility are durable-harness gaps that no release will close. The pre-release tell is what the fix would encode.
- SourceDoes the strategy generalize outside frontier labs, who have privileged visibility into the next model? An external team is betting on a release it can't see.
- ResolvedThe bet depends on a reliable release cadence and a forecastable capability curve (Task Time-Horizon Scaling). What happens to "build for the next model" if model improvement stalls (the stalled-but-diffused future)? Answered: What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful? — it degrades gracefully, because correctly practiced it is a cheap call option on the release cadence: a stall costs the premium (the prototype portfolio expires unexercised), not the firm, provided market validation was kept separate from the capability bet. Three cushions: Latent Capability Overhang keeps effective capability rising post-stall (mine the model replaces wait for the model); the previously-wrong move — engineering the compensating crutch — reclassifies as correct ("too AGI-pilled" correction becomes the standing posture); and competition shifts to the durable layers that never migrated inward. The bet only fails hard under miscalibration: shipping products whose core loop depends on absent capability, which is vaporware whether or not the cadence holds.
- SourceWhen does "generate three and compare" become wasteful — at what decision weight is a real argument (or a design doc) still cheaper than three implementations?
- NowIf design discussion lives in PRs/prototypes, where is the rationale recorded for future readers — does the "why we chose this" knowledge survive, or does it share the staleness problem of Code as Source of Truth? Partially answered: Where Does the Why Live? — it does not survive: discussion buried in merged PRs is orphaned for future readers, and no candidate home holds it durably. What remains open is whether any of the partial patches (richer plan artifacts, a compiled knowledge base) works in practice at team scale.
- Campfire2 open
- SourceCampfire claims its AI edge comes from "our own foundation model." For an ERP, what does a custom foundation model actually buy over fine-tuning a frontier model — and is it durable as frontier models improve (cf. Harness Shrinkage as Models Improve)?
- Wait"Never had anyone outgrow Campfire" — does that hold as customers reach true enterprise scale where NetSuite's breadth historically mattered?
- SourceThe
0/48static and0/29adaptive results are suite- and budget-bounded (40 iterations, a GLM-5.2 attacker, one author's vector corpus). Does the deterministic gate hold under longer adaptive budgets, stateful side channels (DNS rebinding, timing), or multi-agent policy-poisoning — the future work the paper names? Partially answered on the suite half only: NetInjectBench (arXiv 2607.10490,empirical) runs an independently-constructed 80-scenario attack corpus in a different domain (network operations), by a different group, on three different models, and its metadata-aware gate also reaches 0 unsafe actions (0/240 model-scenario instances, 95% Wilson upper bound 1.58%) — plus 100/100 correct decisions on a 90-instance degraded-metadata stress suite. So the zero is not an artifact of one author's vector corpus. The budget half is untouched: that attacker is entirely static (fixed scenarios, temperature 0, no optimizer, no defense-aware adaptation), so it corroborates the0/48static suite and says nothing about the0/29adaptive run. - SourceThe
authzallowlist stops value-redirection but not corruption of legitimately-variable data. Is there a per-call scheme that constrains free-text / open-ended arguments without collapsing utility — or is that irreducibly the province of provenance/data-flow tracking (CaMeL Strict, at ~50pp utility cost)? Partially answered on the parenthetical only: APPA (arXiv 2607.24625,empirical) shows the ~50pp is not intrinsic to flow tracking — branching a restrictive read into an isolated child trajectory instead of tainting the parent recovers most of it (0–26pp of episodes, and zero on the strongest model measured). So "irreducibly the province of provenance tracking" no longer implies "irreducibly expensive." The residual itself is untouched and reproduced a third time: APPA's ownhide-secret-in-statusbreach is a secret smuggled inside an authorized send to an authorized reader, and the authors state that content confinement inside a permitted flow "a label algebra over recipient sets does not claim to provide" — the same class that survives Progent at 22.2% and this page'sauthzstage. Three independent architectures now stop at the same wall. - WaitThe deployment-tier ~3.2× exposure gap (0.603 vs 0.189) means the cheap models chosen for high-volume agent traffic are the most likely to emit the unauthorized call — exactly where a per-call gate is most load-bearing. Does model improvement shrink the attempt rate enough that the gate becomes optional, or is the gate the durable control while models stay jagged? Partially answered: on a different surface — payloads planted in agent memory files rather than model-emitted arguments — Bad Memory (arXiv 2607.14611,
empirical) finds capability does not order the exposure: mean ASR falls with strength inside the Claude family (Haiku 4.5 63.3% → Opus 4.7 30.0%) and rises with it inside the Codex family (GPT-5.2 23.3% → GPT-5.5 60.0%), with the strongest Codex model at 100% ASR on the subtlest goal. Worse for the "gate becomes optional" reading: the most resistant model measured (Opus, 18.3% mean ASR under chaining) is also the most likely to leave the payload in place for a weaker successor (93.3% persistence), so improvement at the top can raise rather than lower system-level exposure. Not a direct answer — this measures neither the paper's frameworks nor unauthorized-argument emission — but it is evidence against tier-based reasoning generally. - SourceOut-of-band policy is load-bearing but under-specified for authoring at scale. The paper forbids any model-sourced policy element; who authors and maintains the verified sets, ceilings, and allowlists for a large tool surface, and does that authoring burden cap the control to high-stakes (money-moving) tools? Partially answered, twice, with opposite answers. APPA (Archestra AI, arXiv 2607.24625,
empirical) supplies a third policy shape: not a central verified set, but a per-tool declared contract — each tool states its own labeldelta, itsemitseffect tokens, and itsrequirespreconditions, and the engine composes them through a lattice fold whose associativity and commutativity are proven rather than tested. Distributing authorship to the tool definition is the answer that plausibly scales, since a tool surface grows one tool at a time. But APPA also supplies the first measured failure of exactly this burden, and it is the sharper data point: in the authors' own evaluation acreate_financetool declared with no sink requirement opened a store-mediated laundering path — write an HR value into finance, read it back under the finance contract — and the paper concedes "prospective enforcement is only as complete as the contracts it evaluates." So the burden does not disappear when you distribute it; it becomes a coverage problem (is every write-side tool declared?) instead of a maintenance problem, and it failed on a fourteen-scenario benchmark with seventeen tools. NetInjectBench's answer runs the other way: in an operations setting nobody authors it — the change-management system already holds it. Its six trusted fields (approval status, maintenance window, approved tool, approved device, approved patch, change-request ID) are the schema of an existing ITSM/CMDB record, so the gate consumes an out-of-band channel the enterprise maintains for its own reasons. That reframes the burden as integration rather than authoring, and suggests the answer is domain-shaped: where a change-control system of record already exists, policy is free; where it does not, the authoring problem stands. Weak as evidence — the benchmark governs two tools, so it never encounters the scale the question is about, and the record is a benchmark field rather than a live system. Escalated, not answered, 2026-08-04: Rashidi's SoK (balkanization execution security research,empirical) makes this its Gap 4 and finds the field-wide absence — Datalog reference monitors, capability tokens, deterministic pre-action gates, information-flow graphs, "every one of them assumes the policy itself is correctly specified by a trustworthy author and asks only whether that policy is then enforced. None studies what happens when the policy is wrong, overly permissive by mistake, internally contradictory, or where a policy author under time pressure grants broader scope than intended because narrower scoping is more work." So the question is not under-answered in this vault by accident; no paper in a 39-paper execution-security corpus measures it. The survey rates it the most consequential of its first four gaps and supplies the argument for why: ShellSieve's 69–98% fragility is measured against denylists real developers wrote and shipped, so policy-authoring error is plausibly at least as large a source of real-world risk as enforcement failure — and it is the one stage of the pipeline nobody measures. The experiment it asks for is well-specified: a ShellSieve-style empirical study aimed at the access-control policies this literature proposes rather than at command denylists, to tell the field whether its mechanisms are undermined more by weak enforcement or by policies never correctly specified. Note this also outranks the two partial answers above: APPA's undeclared-create_finance-contract breach is an instance of policy-authoring error caught in the wild, which the survey's framing predicts should be common and unmeasured.
- SourceThe
- SourceThe >95%/<5% figures are session-level; what's the false-positive rate for legitimate security researchers and biologists, whose benign queries are exactly the ones most likely to trip the conservative classifiers? Partially answered: on FrontierBench (74 hard science/engineering terminal tasks), Claude Opus 5's classifiers flagged 5% of API calls in 4% of trials where Fable 5's flagged 42% in 26% — so the over-broad tuning was costing roughly a quarter of trials on exactly this kind of legitimate technical work, and has been substantially narrowed. Still a benchmark proxy, not measured professional traffic. Further evidence (2026-07-30): a competitor's benchmark runs (Kimi K3 card) put Fable 5's fallback rate at 35% of SWE-Marathon tasks, 17.5% of Kimi Code Bench tasks and 40% "downgraded" on Agents' Last Exam — corroborating the order of magnitude from outside Anthropic, and locating it on plain software engineering rather than only on science-adjacent work. Still benchmarks, still not professional traffic.
- NowFallback-not-refusal preserves UX but means the real general-access model for security/bio-adjacent work is Opus 4.8, not Fable — does that quietly cap Fable's value for whole professional segments until the trusted-access programs open? Partially answered (anecdote, 2026-07-24): Cline abandoned a 17-hour autonomous evals-research campaign on Fable 5 because the classifier "kept downgrading the model to Opus-4.8," and ran it on a competitor's model instead — the first instance in this corpus of the cap being paid as a lost workload rather than as a lower benchmark score, and from outside Anthropic. One vendor's passing remark with no rate attached; it establishes the failure mode exists in the wild, not its frequency.
- SourceThe UK AISI's "progress toward a universal jailbreak" is disclosed but not quantified — and the post-launch access suspension (see Claude Fable 5) raises the question of whether a safeguard failure forced it.
- SourceDoes swapping to a weaker model on flagged topics create an exploitable oracle (probe which queries trigger fallback to map the classifier's boundary)?
- SourceHow is character versioned across model releases? Public commentary doesn't show change-logs at character level.
- SourceCould character be reproduced by competitors via fine-tuning, or is it path-dependent on Anthropic's internal practice?
- SourceFor non-coding products like Cowork, does the same character work, or does Cowork need its own character tuning?
- Claude Code Auto Mode4 open
- SourceWhat false-positive rate does the classifier have on routine-but-aggressive refactors (e.g., large-file renames,
rmof build artifacts)? - SourceHow well does the classifier generalize to custom tools / MCP servers where it lacks environment context?
- SourceIs the classifier's decision boundary documented/stable enough for security-sensitive orgs to certify, or is it effectively a black box whose behavior drifts with updates? Partially answered (2026-08-04): the changelog supplies the two facts a certification argument would start from — the classifier defaults to Sonnet 5 for external sessions, and it is "validated on the session's first request and pinned for the session" (v2.1.210), so the boundary cannot move under a running agent. That closes intra-session drift and closes nothing else: no decision boundary is documented, the scope of what the classifier adjudicates widened in v2.1.218 (taking over the dangerous-rm, background-
&, suspicious-Windows-path and unprovable-read-only-Bash checks from static analysis), and release-to-release drift is unaddressed and now covers more surface than before. - SourceDoes extending auto mode to API users change its calibration — is the classifier retrained for automation-heavy use, or held constant?
- ResolvedCompared to OS-level sandboxing (mentioned in Claude Code Best Practices alongside auto mode), what's the defense-in-depth story? When should both be layered? Answered: Classifier Gates vs OS Sandboxing: The Defense-in-Depth Story for Auto Mode and Cowork — they are different control kinds with disjoint blind spots: the classifier is a semantic gate (judges intent the sandbox can't see — within-capability harm over allowed channels) and the sandbox is a structural barrier (contains the classifier's two documented failure modes). They fail independently by mechanism, making the stack real defense-in-depth rather than correlated friction. Layer both whenever the agent holds reach beyond the sandbox boundary (live creds, MCP to real SaaS), runs unattended, or reads untrusted input; sandbox-only is legitimate when containment is total (the Hermes Agent container-is-the-boundary design point); classifier-only is a stopgap for interactive low-stakes local work — never for unattended runs.
- SourceWhat false-positive rate does the classifier have on routine-but-aggressive refactors (e.g., large-file renames,
- SourceDoes the instruction-count ceiling hold for conditional policy, where only a handful of rules bear on any given turn? Every rule in Eliav's experiment applies to one generation simultaneously; a CLAUDE.md is mostly situational ("when editing migrations, …"), so N=80 is a floor on the harshest possible loading and says nothing about a 200-rule file of which five fire per turn. Falsifiable directly: hold the applicable subset fixed and grow the inapplicable remainder.
- SourceHow does the Writer/Reviewer pattern compare to agent-to-agent review (as in OpenAI's Codex workflow)?
- SourceWhen does subagent overhead exceed the benefit of context isolation? Partially answered 2026-08-03 by codex 0 to 10m users chatgpt work (
practitioner-opinion, no measurement) — a task-shape criterion rather than a crossover point. Akshay Nathan (OpenAI): multi-agent modes "are best for when you have tasks that are either incredibly complicated, like open explorations, or very paralyzable… but for most tasks, they don't fall into either of those buckets," so the default should be a single agent. Note the overheads he actually names are neither context nor tokens: rate-limit consumption (Ultra "can use more of your limits," which is why OpenAI moved it behind advanced settings post-launch) and human legibility (sub-agent transcripts hidden by default to avoid overwhelming users — see Shared Harness, Differentiated Surfaces). A practitioner counter-practice in the same episode pulls the other way: Vibhu reports telling every long-running task to "use sub-agents where possible" for wall-clock and for cost, fanning out to cheaper models — which Cost-per-Task Over Cost-per-Token argues is the wrong default. Second instance (2026-08-04): Willison runs the same practice with the tier choice itself delegated — "for all coding tasks use your judgement to decide an appropriate lower power model and run that in a subagent" — and reports only that his Fable allowance shrinks more slowly. Two practitioners now default to the fan-out; neither measures it. What remains open is the measured crossover, which no source in the corpus supplies. - ResolvedWhat's the optimal CLAUDE.md length before instructions start getting lost? Is there a measurable threshold? Answered 2026-08-04 by prompt design at scale (Eliav, arXiv 2607.19257,
empirical) — full treatment at Instruction Compounding. Yes, there is a threshold, and it is count-shaped rather than length-shaped: across five models (including Claude Sonnet 5 and Haiku 4.5), the rate at which every instruction in the prompt is obeyed falls steeply by N≈40 simultaneous verifiable rules and hits zero by N≈80, holding through N=160 and identical across markdown, plain text, prose, and table renderings and across system-prompt vs. user-turn placement. The paper's own prescription is the answer in usable form: 40 simultaneous instructions is a redesign point, not a tuning point — past it, splitting across turns, tools, or a validation pass is the only thing that works, and reformatting is not. This also settles the residual the 2026-08-03 retag left behind (does aggregate size have an independent effect once each line passes ablation?): yes — every rule tested was distinct, non-redundant, and individually satisfiable, so per-line ablation non-inferiority would have cleared all of them and still missed the collapse. Two scope limits carried on the answer: "perfect response" is a strict conjunction, so some of the floor is the arithmetic of ANDing N checks rather than the model dropping the block, and every rule tested is a hard output constraint applied to a single generation. The conditional-policy case a real CLAUDE.md actually presents is now the successor question in Open Questions above.
- Claude Design2 open
- WaitDid the "any design tool via MCP" integration actually ship on the stated timeline? (Forward claim from May 2026.) Partially answered: Nate Parrott's July 2026 post confirms "web search and MCP connections work in Claude Design too, whenever the design depends on outside information" — the client side of the claim is live. Whether specific design tools integrate through their own MCP servers is still unconfirmed.
- SourceHow does Claude Design's eval discipline work for visual/aesthetic output, where there's no compiler or test? (Same open question as Cowork for non-code artifacts; relates to character/taste evals.)
- Claude Fable 53 open
- SourceWhy was access suspended after launch? The source banner gives no reason (capacity? a safety finding? the UK-AISI jailbreak progress noted in Capability-Gated Model Fallback?). Not in source.
- SourceExact benchmark numbers vs GPT-5.x / Gemini are image-only in the source; not transcribed.
- SourceHow much of Fable's general-access experience is actually Fable vs Opus-4.8 fallback for security-research-adjacent users whose queries trip the conservative classifiers?
- Claude Mythos 53 open
- SourceSuspension reason — shared with Fable 5; not stated in source.
- SourceHow does "somewhat stronger than Mythos Preview" square with Opus 4.8's card claiming Mythos Preview was the capability frontier? The frontier has moved; the magnitude isn't quantified here.
- WaitThe bio trusted-access SKU is "Fable 5 with bio safeguards removed," not Mythos 5 — so "Mythos 5" strictly denotes the cyber-lifted variant. Whether these converge under one trusted-access umbrella is unstated.
- Claude Opus 4.75 open
- SourceDo Hakim's (2026) brevity-constraint findings on Opus 4.6 replicate on Opus 4.7, or does the literal-instruction-following change the elasticity? Specifically: does
<50 wordsstill yield +13.1pp on GSM8K? - SourceDoes Opus 4.7 still underperform as a planner in HotpotQA-style combo sweeps, or does improved instruction-following close the gap that AgentOpt (Hua et al., 2026) identified?
- SourceWhat is the real-world token-inflation multiplier on typical Claude Code sessions (1.0–1.35× is content-dependent — what's the distribution on code-heavy vs. prose-heavy inputs)?
- SourceHow does xhigh compare to max on coding evals? The migration guidance says "start with high or xhigh" — is max ever worth it for coding?
- SourceWhat fraction of existing CLAUDE.md / system-prompt hedges become counterproductive under literal instruction following?
- SourceDo Hakim's (2026) brevity-constraint findings on Opus 4.6 replicate on Opus 4.7, or does the literal-instruction-following change the elasticity? Specifically: does
- Claude Opus 4.82 open
- SourcePublic model ID and pricing: the card does not state them; presumably
claude-opus-4-8at the Opus tier. - SourceWhy is 4.8 less robust to prompt injection than 4.7 despite broad alignment gains — a capability/robustness tradeoff, or an artifact of the eval surface? Partially answered: Claude Opus 5 reverses the regression by an order of magnitude on the same adaptive-attacker suites, ruling out a durable tradeoff but leaving 4.8's own dip unexplained — Anthropic does not return to it.
- ResolvedDoes the grader-speculation trend continue to escalate in the next model, and at what point does it begin to affect outward behavior? Answered: Claude Opus 5 (July 2026) — it did not escalate in verbalized form; Opus 5 verbalizes evaluation awareness less than Sonnet 5, Opus 4.8 or Mythos 5, while NLA readouts show internal grader modeling as the largest cluster and UK AISI measures prompted eval-vs-deployment discrimination rising to 95%. On the outward-behavior half: one documented instance (an undisclosed
curlrule violation with NLA-decoded grader speculation and internal "deceptive" representation), inside a class measured at under 0.01% of monitored completions. The generation-tracking version of this question now lives on Evaluation Awareness & Grader Gaming.
- SourcePublic model ID and pricing: the card does not state them; presumably
- Claude Opus 53 open
- SourceAnthropic says the origin of the fall in verbalized evaluation awareness "is unclear." Is it genuine, or has the awareness simply become harder to verbalize while remaining internally represented?
- WaitThe card does not report a bug bounty for Opus 5 (still being set up at publication). Do live human red-teamers reproduce the near-zero adaptive-attacker success rates?
- WaitAnthropic commits to covering multi-agent and subagent settings in future assessments after Mythos 5's review flagged that Opus 5 "can relay claims from subagents to users without verifying them." Does that relay failure survive into the next generation?
- Claude Sonnet 54 open
- SourceThe head-to-head benchmark numbers vs Sonnet 4.6 and Opus 4.8 are image-only in the source; the System Card has the full set.
- SourceWhat is the real-world token-inflation multiplier on typical Sonnet 5 traffic (1.0–1.35× is content-dependent), and does "roughly cost-neutral" hold once effort levels rise?
- SourceWhy does a mid-tier model show higher behavioral-audit misalignment than the more capable Opus 4.8 and Mythos Preview — a capability-alignment coupling, or a training-recipe difference between the Sonnet and Opus/Mythos lines?
- SourceAt what effort level does Sonnet 5 actually match Opus 4.8, and how does the crossover cost compare to just running Opus 4.8? Partially answered: Anthropic's model-selection guidance reframes the crossover as a topology choice rather than a point on the effort dial — Sonnet 5 with a Fable 5 advisor reaches within 10% of Fable 5 on SWE-bench Pro at 63% of the cost. That is a different pairing (Fable, not Opus 4.8) and a different mechanism (selective coaching, not raised effort), so the effort-dial crossover itself is still unmeasured. Second half answered (2026-08-04): on Databricks' internal coding bench the crossover cost comes out unfavorable — $2.09/task at 81% success versus Opus 4.8's $1.94 at 87%, so running Opus 4.8 was cheaper and better on that workload. The effort level is not reported, so the first half — at what effort Sonnet 5 matches Opus 4.8 — remains unmeasured, and one bench on one company's codebase does not generalize.
- SourceHow does combination-level optimization interact with continual model releases? If Claude Opus 4.7 ships next month, does the full Pareto frontier need re-running, or do warm-started bandits adapt cheaply? Partially answered (2026-08-04, by synthesis): What Makes a Self-Improvement Artifact Transfer? — a combination is a solver-fitted artifact (fitted to the current menu's capabilities and prices), so the frame predicts full re-runs rather than cheap adaptation, with the durable residue being assignment rules rather than assignments; the HotpotQA→Cursor inversion below already shows the assignment not transferring while the reconciling rule does. The warm-started-bandit half is an empirical question no source measures.
- SourceAt what pipeline depth does the combinatorial search become intractable even for Arm Elimination? The paper tests up to ~81 combinations; production pipelines with 5+ roles and 10+ candidate models each blow past that.
- SourceDoes the "weak planner + strong solver" pattern generalize, or is it specific to HotpotQA's delegation dynamic? Recommender-critic, drafter-editor, and retriever-generator topologies might invert. Partially answered — it inverts (2026-08-03): on Cursor's long-horizon build task the efficient frontier is the opposite assignment, strong planner + cheap worker, with the entire worker fleet costing $411 under an Opus 4.8 planner versus $9,373 when a frontier model did both jobs at the same quality. The reconciling variable is what the planner is able to do: HotpotQA's planner could answer directly and did; Cursor's cannot. So the pattern is specific to the delegation dynamic, and the general rule is about foreclosing execution at the planner role rather than about weakening the model in it.
- SourceWhat's the right way to re-evaluate when the tool environment changes? AgentOpt assumes fixed tools — adding or removing a tool potentially invalidates the whole frontier.
- SourceIs there a cheap per-call classifier that can predict which combination will win on a given query, avoiding combo-level evaluation entirely? Sharpened (2026-08-03): Writer's harness swap proposes classifying on feature demand rather than difficulty — which orchestration features (delegation, MCP tool use, multi-step workflows) a request will exercise — on the evidence that those features carry capability floors and that a model below the floor fails on them regardless of how simple the prompt reads. That is a candidate classifier target, not a classifier: nobody has built or evaluated one.
- Code as Source of Truth2 open
- NowWhat knowledge genuinely can't live in the codebase (org strategy, the "why," cross-team context) and therefore still needs a durable doc — and how do you keep that small slice current? Partially answered: Where Does the Why Live? — the "why" is the clearest such slice, and every candidate home fails or only partly works; a compiled knowledge base outside the staling code is the least-bad option. The "keep it current" half is untouched.
- SourceIf onboarding is "ask Claude," what happens to the tacit knowledge that was previously transferred socially in deep-dives — is it captured anywhere, or quietly lost?
- SourceIs there a public schema registry so external orchestrators can target specific App Server versions without
generate-json-schema? - SourceThe "dynamic tool calls (experimental)" caveat — what's the stability roadmap? Symphony depends on this for its security model.
- SourceHow well does the protocol handle multi-modal turns (image inputs, screenshot attachments)? The spec is text-focused.
- ResolvedHow does the App Server protocol compare in detail to MCP? Both expose tools to a model, but App Server is inside the Codex runtime while MCP is outside. When does each win? Answered: App Server vs MCP, and the Claude-Side Equivalent: Three Boundaries for Driving Agents — they sit at different planes and mostly compose rather than compete: MCP is the model↔world tool plane (write once, consume on every surface, provider-operated), App Server is the orchestrator↔runtime session plane (thread lifecycle, turns, events, timeouts, token accounting — none of it in MCP's scope). The only overlap is dynamic tool calls, where the rule is: MCP for reusable cross-surface third-party capabilities; orchestrator-injected tools for session-scoped, credential-sensitive ones (the
linear_graphqltoken-isolation pattern, which also shrinks the poisoned-metadata/rug-pull attack surface to first-party code) — at the cost of experimental stability and zero ecosystem reuse. - ResolvedIs there an analogous protocol on the Claude side, or is Claude's equivalent exclusively the Agent SDK + tool-use API? Comparing the two would clarify when "drive an existing CLI" beats "build on the SDK." Answered: App Server vs MCP, and the Claude-Side Equivalent: Three Boundaries for Driving Agents — no documented Claude-side protocol; the offering brackets the App Server's position:
claude -p(drive the product, inherit the full harness — permissions with abort-don't-hang unattended behavior, skills, context files, MCP wiring — but get text, not structured events) and the Agent SDK (build a different product on the raw runtime — Claude Design's weekend prototype). Rule: drive the CLI when the product's harness is the value and orchestration is batch/fan-out shaped; build on the SDK when the agent is a different product with its own surface and tools. Symphony's own tmux→protocol evolution marks the middle layer (structured session control over the product harness) that the Claude side currently approximates from either end — whether it gets standardized or harness shrinkage makes it moot is the watch item.
- SourceIs there a public schema registry so external orchestrators can target specific App Server versions without
- SourceThe design cannot separate "AI adoption improves team social health" from "healthier teams adopt AI better", and the authors say so. The discriminating study is the one they name: longitudinal or quasi-experimental tracking of AI adoption and team dynamics over time, controlling for communication culture, org maturity, leadership practice and seniority. Until then every coefficient here is an association.
- SourceThe Information Sharing path — AI adoption directly worsening documentation and information governance — is the only harm signal in five models and lands at p = .069, below the β ≈ .20 the sample can reliably detect. Does it survive at n ≈ 400, and does it strengthen in teams without a documentation discipline? This is the falsifiable half of the paper's "governance-dependent" conclusion.
- SourceThe study measures peer-interaction frequency, not what the interaction contains or what expertise is retained. If AI raises the count of specialization-oriented exchanges while lowering their depth (the comprehension register), frequency would rise exactly as measured while the underlying transactive memory thins. Nothing here distinguishes the two.
- Compounding Data Moat4 open
- SourceIs the "two-year replication window" claim defensible empirically, or aspirational? The playbook does not cite measurement. (Partially answered, at the far end only: DroneDeploy is a ~decade accumulation whose value arrived discontinuously when vision models did, priced at $845M — one case, told by its own investor, with no counterfactual for how fast a late entrant could have caught up once the demand was legible. What it suggests is that "two years" is the wrong unit: the binding variable is whether accumulation started before the monetizing capability was foreseeable, not elapsed calendar time. A real answer still needs a matched pair — two vertical products, one with a pre-capability archive and one without, competing after the capability lands.)
- WaitHow does this moat hold up when foundation models themselves continue improving rapidly? If a generalist model in 2027 has internalized enough vertical context to handle 340B drug claims natively, does the vertical-edge-case moat erode?
- SourceThe data-flywheel argument has been made for SaaS for 15 years. What's actually different in the AI-native version? Probably: the data improves the model in addition to the product, but the playbook doesn't make this distinction precisely.
- SourceThe "customers build APIs on top of you" lock-in is structurally similar to platform plays (Salesforce AppExchange, Shopify apps). Is the moat type really new, or just newly accessible to lean startups?
- SourceThe loop assumes the team is (close to) the user. How much of the compounding advantage survives when the user is unlike the builder and "talk to users" can't be same-room?
- SourceWhere is the line between worthwhile internal tooling and yak-shaving? Carey's "afternoon" bar is the heuristic, but Cat Wu warns that over-customizing setups "becomes distraction."
- SourceDoes Claude-as-first-pass-on-all-feedback ever filter out the rare signal that doesn't cluster? Automating triage optimizes the common case; the tail is where surprising bets come from.
- Compute Allocator2 open
- SourceIs 1% a Thariq-specific number or a regime? For larger, more code-heavy projects the production residue is presumably higher; what sets the ratio?
- SourceAllocation quality is hard to measure — what's the feedback loop that tells an allocator they spent compute badly (vs. just spending a lot)?
- ResolvedDoes treating humans as "compute allocators" risk the oversight-fatigue / accountability failure modes the HBR research flags, where the human nominally decides but actually rubber-stamps? Answered: Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? — yes, it is the role's central failure mode, documented at three evidence layers (brain-fry error rates +11%/+39% and the under-engagement swap; Faros's 31.3% no-review telemetry; the CMU theory's P1–P4 load-and-plausibility mechanisms), with two allocator-specific aggravators: allocation quality has no feedback loop, and the human's retained 70% planning share is exactly where rubber-stamping is transcript-invisible. The framing survives only with structural countermeasures — understanding-gated merges (quiz gate), sample-based depth concentrated on high-stakes points, risk-tiered gating — that make understanding rather than signature the merge condition.
- SourceCan you certify "no benchmark-maxxing" — verify a reported score used a stated, reproducible compute budget rather than a hidden best-of-N scaffold?
- SourceCompute has several units (tokens, dollars, wall-clock). They diverge (a more efficient model wins on cost but not always on tokens). Which x-axis is the honest one, and does it depend on the buyer? (AISI reports against tokens on a log axis, and notes that as cost-per-token falls, the high budgets that reveal capability become progressively cheaper to reach.) Partially answered (2026-08): METR argues dollars, and supplies the measurement that makes the argument bite rather than assert — on an agentic AI R&D task, experiment compute is ~70–90% of trajectory cost, so tokens and dollars are not proportional and a token axis omits most of the spend. It also names the buyer the axis is for (a lab choosing between human and agentic labour, which only dollars can price) and demonstrates the payoff: a second, human curve can be drawn on a dollar axis and cannot be drawn on a token one. Still open in two respects — the finding is one task class (long-horizon optimization with heavy experiment compute; a chat or single-shot benchmark inverts the ratio), and the dollar axis buys comparability at the cost of importing a wage assumption and the evaluator's own harness efficiency.
- Does a compute-controlled evaluation regime advantage frontier labs (who can afford the full curve) over academics and third-party evaluators who can't? Sharpened (2026-07): a government evaluator (AISI) does run the full curves — so it is affordable to a well-funded public body — but AISI itself flags that "the most informative evaluations may be expensive" and is researching how to forecast high-budget performance from cheap runs precisely to relieve that cost. So cost is the binding constraint even for a funded third party; it just isn't fatal to one. Partially answered on a sibling axis (2026-07): BenchPress shows the analogous cost problem on the benchmark-count axis is largely solvable — a model's full 133-benchmark scorecard is recoverable from ~5 probes to within ~3.93 points — but that reduces which benchmarks to run, not the compute-per-benchmark the full-curve question is about, so it relieves eval cost on a different axis than the one this question poses.
- SourceGemma 4 controls for compute in its long-context table and not in its headline table, without comment. Is partial control worse than none — does it lend the uncontrolled tables borrowed credibility?
- SourceIs a grid with a price row but no token counts more misleading than a grid with no cost information? Falsifiable directly: run the four models in DeepMind's table on one agentic benchmark, record tokens-per-task, and check whether the price-implied cost ranking survives.
- SourceMoonshot reports Fable 5 hitting fallbacks on 35% of SWE-Marathon tasks and 40% of Agents' Last Exam tasks "downgraded". Would re-running those with safeguards disabled change the ranking — and should a leaderboard publish the safeguarded score, the unsafeguarded one, or both?
- Confident But Unsure3 open
- SourceIs the +11% accuracy / +6% hallucination pairing an inherent consequence of lowering the abstention rate, or are they separable with calibration training?
- SourceThe silent 0.85 → 0.82 substitution has no proposed mechanism in the card. Is it the same phenomenon as guessing, or a distinct output-layer effect?
- WaitAnthropic committed to building new overconfidence metrics. Will they reproduce the observational finding, or saturate like the three existing diligence evals?
- SourceThe "human" is an LLM simulator (GPT-4.1) that also judges — how much of the configuration-dependent structure is a property of human-agent collaboration versus an artifact of GPT-4.1 modeling both sides? The Appendix C.6 simulator-swap is the only check, on a subset.
- SourceA4 is instantiated as a fixed one-shot proactive intervention. Real proactive humans time their input adaptively — does the "premature/distracting intervention breaks tasks" result hold, worsen, or vanish under a human who chooses when to interject?
- SourceThe optimal channel is shown to be pattern-specific, but pattern is an oracle label assigned at construction. Can an agent infer which pattern it is in (and thus which channel to solicit) at runtime — the actual deployed skill the paper says agents lack?
- WaitDoes the "more channels adds coordination overhead" penalty shrink as the backbone improves (a capability gap), or is it a structural cost of mixed-initiative interaction that persists?
- SourceDoes the asymmetry regenerate faster than it transfers? The whole human role, under this frame, rests on the answer. Nobody in the corpus has posed it.
- SourceNg prefers the frame because it "gives us a clearer path to helping AI systems get better." That is a reason to adopt the frame, not evidence that it's true. What would distinguish a context asymmetry from a capability gap empirically? (Returns to Expertise in Agentic Coding is the closest thing to an instrument.)
- SourceIf the human's contribution is context injection, is the human replaceable by better context plumbing — memory, retrieval, continuous production telemetry — rather than by a better model? That would put the expiry of human-in-the-loop on the infrastructure roadmap, not the scaling curve.
- SourceNg writes from 0-to-1 consumer products. Does the frame survive contact with domains where the missing thing is a concept rather than a fact?
- SourceDoes the input-token reduction survive a matched billed-cost audit once side-channel planner calls and prefix-cache breaks are charged? The paper reports prompt-surface impact and explicitly declines this claim. Partially answered (2026-08-03): Prompt-Cache Economics supplies the audit for the class but not for Self-GC — CAPC reconciles measured API spend against Anthropic's invoice to within 1% and finds a token-reducing technique (query-aware compression, 3× fewer tokens) costing +40.1% more than sending nothing compressed on τ-bench retail. So the concern the question encodes is real and measured; a Self-GC-specific billed-cost audit still doesn't exist. Annotated (2026-08-04): agentic context management supplies an analytic bound on one half and explicitly declines the other. The overhead half: because each compaction pass operates on the already-compacted context plus recent turns, the number of passes grows only linearly and total cost is
N·W·(1 + c/p)— a fixed multiplicative factor rather than a growing tax (~1.25× at the paper's illustrative p = 8, c = 2). If Self-GC's side-channel planner call behaves the same way, planner overhead cannot asymptotically eat the savings. The cache half is dropped by assumption ("ignore caching discounts, which shift the constants but not the asymptotics"), which is precisely the term the question is about — so this narrows the question to the cache break alone and answers none of it with a measurement. - NowIs the 0.3 expected-pruning break-even for immediate commit portable, or a function of one provider's cache pricing and TTL? Stated as an operating policy over one deployment's regression. Partially answered (2026-08-03): Prompt-Cache Economics settles the portability question in principle — the analogous threshold is ρ_cross(r) = (α − 1/r)/(α − β) over the write premium and read discount alone, and it moves sharply with both provider and TTL (α = 1.25 on Anthropic's 5-minute cache vs 2.0 on its 1-hour cache; OpenAI α = 1.0, β = 0.5). Read across, the number is not portable and the derivation is. Re-deriving 0.3 under that parametrization is now a synthesis over pages already in the wiki.
- SourceHow does object-level GC compare against clear-and-restart on the same traces? No source in the corpus measures the two against each other, and they optimize different things — GC preserves in-run dependencies, clearing resets attention quality.
- SourceHas anyone other than a vendor measured that validated compaction actually preserves fidelity where crude summarization does not? The whole three-regime argument turns on a cell — linear cost with checked fidelity — that no source in the corpus has isolated. Maximem's 92.0%/93.2% are end-to-end system scores on conversational-memory benchmarks with no compaction ablation, its validation mechanism is undisclosed, and Self-GC's no-impact judge grades candidate plans offline rather than a returned validation score. The falsifiable form: an A/B of the same compactor with the information-loss check on and off, on the same traces, scored on downstream task success.
- SourceDoes the "no semantic interference from coordination volume" null survive at the prompt lengths agents actually run? RCWT's intact-task ablation holds at ceiling to a 0.95 coordination ratio, but its largest condition is ~14,000 total tokens — while the sessions this page's other sources measure average 70–90k input tokens per request. The falsifiable form: rerun the intact-task ablation with the same 698-token task block at 100k, 250k and 500k of surrounding coordination content, on models whose per-model effective ceilings Context Window Smart Zone shows are not predicted by the advertised window. If the null holds there, displacement is the whole story; if it breaks, there are two mechanisms and the corpus has been attributing both to one.
- SourceWhat fraction of agent memory failures are reasoning-sufficiency failures rather than retrieval failures — the bridge document missing while a relevant document was returned? Every memory benchmark in the corpus scores a hit against a single gold target, so the quantity is structurally unmeasurable by all of them; the only bearing datum is that LongMemEval's multi-session category (75.2%) carries nearly all of one system's residual error. Named trigger: Maximem states a benchmark measuring accuracy, latency, token efficiency and context-rot resistance together is forthcoming.
- SourceDoes the input-token reduction survive a matched billed-cost audit once side-channel planner calls and prefix-cache breaks are charged? The paper reports prompt-surface impact and explicitly declines this claim. Partially answered (2026-08-03): Prompt-Cache Economics supplies the audit for the class but not for Self-GC — CAPC reconciles measured API spend against Anthropic's invoice to within 1% and finds a token-reducing technique (query-aware compression, 3× fewer tokens) costing +40.1% more than sending nothing compressed on τ-bench retail. So the concern the question encodes is real and measured; a Self-GC-specific billed-cost audit still doesn't exist. Annotated (2026-08-04): agentic context management supplies an analytic bound on one half and explicitly declines the other. The overhead half: because each compaction pass operates on the already-compacted context plus recent turns, the number of passes grows only linearly and total cost is
- SourceDoes the smart-zone marker scale with model size, or is it bounded by attention architecture? Pocock observes "the dumb zone has become less dumb lately" but pegs it at 100K through 2026. Partially answered 2026-08-04 by prompt design at scale (
empirical) — neither cleanly, because the premise of a single marker doesn't hold. Degradation onset is a per-model effective ceiling that the advertised window does not predict: two models sharing the same documented 1,000,000-token ceiling diverge sharply in format-spread growth over the identical 256k→512k range. It also splits the question by task — retrieval holds at 0.98–1.00 through 64k and past 128k for some models, well beyond the 100K marker, which is consistent with Pocock's own retrieval-vs-reasoning carve-out. The paper measures no reasoning task, so whether the ~100K reasoning marker is architectural remains untested. - WaitWhen sparse-attention or memory-augmented architectures ship, does the smart zone become a soft constraint?
- SourceHow should harnesses surface remaining smart-zone budget to the user — token count, percentage, or a richer signal?
- SourceDoes the smart-zone marker scale with model size, or is it bounded by attention architecture? Pocock observes "the dumb zone has become less dumb lately" but pegs it at 100K through 2026. Partially answered 2026-08-04 by prompt design at scale (
- NowHow much of the +12% is controlled variance in information collection versus the removal of the interviewer's discretion to abort? Human recruiters screen-out mid-interview 25% of the time against the AI's 7%, which mechanically suppresses human-arm offers before the evaluation stage. Falsifiable: re-estimate the treatment effect on the subsample of interviews that reached completion in both arms, or instrument the screen-out decision. The paper reports both numbers and never decomposes them.
- SourceThe AI system is never identified — no model, vendor, or version, only that Google Cloud supplied infrastructure. Is "controlled variance" a property of a 2025-generation voice agent under this firm's prompt, or of AI-conducted interviews generally? Nothing in the paper lets a replication know what it is replicating.
- SourceRetention ≥1 month is both the quality proxy and the metric the recruiting firm is paid on by its clients. Does the AI advantage survive on an outcome the intermediary is not compensated for — client-side performance at 12 months, promotion, or wage growth? The four-month estimate, the longest horizon measured, already fails significance under recruiter clustering.
- Conversation Artifacts3 open
- SourceTokens are a proxy for both compute cost and output value, but verbose models inflate tokens per unit of intent (the same critique Conversation-to-Delegation Shift raises); how much of "compute tracks value" is genuine value vs. models simply emitting more?
- SourceThe reading-level "+1 year" gap may be register (terse prompts, polished replies) rather than substance; can it be separated from genuine elevation of content?
- SourceArtifact classification is first-party and single-model-graded; do the 30+ categories and the work/personal/coursework split survive independent replication?
- SourceThe token-share metric rewards verbose agentic output. How much of the 99.8% / 63.3% / 16.5% spread is a genuine work shift vs. agentic tools simply emitting more tokens per unit of human intent?
- SourceOpenAI-internal is a frontier preview by assumption. Does the external organizational curve actually trace the OpenAI path (the paper's implicit claim), or does it plateau where adoption frictions don't vanish?
- Source"Asking is half of ChatGPT, doing is most of Codex" — but the two tools self-select different work. How much of the asking→doing contrast is the shift itself vs. routing pre-existing "doing" tasks to the tool built for them?
- SourceDoes "cost-per-task is lower for more intelligent models" survive measurement on non-Anthropic production traffic? The claim is stated without data and the published curves are explicitly illustrative. Still open, with a near-miss (2026-07-30): DeepMind's Gemini 3.5 Flash-Lite card is a non-Anthropic instance of the shape — +67% output price, a full agentic tier of capability — but reports no tokens-per-task, so it supplies the premise and not the measurement. Partially answered (2026-08-03), and it splits: Cursor's four model mixes are the measurement — non-Anthropic infrastructure, a real four-hour workload, matched time budgets, matched quality, published dollars. Within the planner role the claim holds: the more expensive Fable 5 planner billed slightly less than Opus 4.8 at roughly twice the per-token price, because it emitted far fewer planning tokens. At the level of the whole run it fails: the same Fable configuration came out substantially more expensive because its workers burned several times the tokens, and the most expensive run of all was the strongest model used throughout ($10,565 versus $1,339). So the thesis appears to be a claim about a role, not about a system, and no source yet measures it on a single-agent workload outside Anthropic. Sharpened, with the first counter-datum (2026-08-03): Writer's harness swap publishes per-model cost and per-model quality for six models on non-Anthropic infrastructure under one pinned price table, and cost per task rises monotonically with model strength on that workload — quality per dollar is worst for the two strongest models (Palmyra X6 3.16, Sonnet 4.6 3.27) and best for the cheapest (Qwen 3.6 4.44). It is a controlled bench rather than production traffic, the arms are not iso-quality, and the capability spread is only eight points, so it does not close the question — but the sign is wrong for the vendor guidance and the paper's own conclusion is that the model menu is the smaller lever anyway. Closest yet, and it splits again (2026-08-04): Databricks' internal coding bench — real engineering tasks on its own multi-million-line codebase, measured by neither Anthropic nor a model vendor — puts Opus 4.8 at $1.94/task and 87% success against Sonnet 5 at $2.09 and 81%, on tokens ~1.7× cheaper. Within Anthropic's own line the claim therefore holds, in the long-horizon coding regime where its mechanism should be strongest, and this is the first time it holds on a third party's real codebase. Against the wider menu it fails: open-weight GLM 5.2 is statistically tied with Opus 4.8 on quality at $1.28/task. So the surviving form of the rule is about tokens-to-completion, not about price tier — a cheaper model is dearer per task when it burns more tokens and finishes less often, which is contingent, not structural. Still not closed:
case-studysecondary reporting, production-derived tasks rather than production traffic, and no n, variance or per-arm methodology behind "statistically tied." The article's own cited generalization (arXiv 2603.23971 — a third of comparisons invert; Gemini 3 Flash 80% cheaper listed, 38% dearer in practice) is the paper most likely to settle this and is not yet ingested. - SourceDoes the advisor strategy's result (within 10% of the advisor's score at 63% of its price) generalize beyond SWE-bench Pro and the Sonnet-5/Fable-5 pairing — and where is the crossover at which advisor calls cost more than they save?
- SourceIs "start with the strongest model" safe inside multi-role pipelines, given AgentOpt's finding that the strongest model was the worst planner? Anthropic's Sonnet-for-sub-agents note hints at a boundary it never states. Partially answered (2026-08-03): Cursor's production swarm says the safe form of the rule is positional — strongest model as planner, cheapest capable model as worker — and that running the strongest model in every role is the single most expensive way to reach the same grade. It also dissolves the apparent conflict with AgentOpt: Opus was the worst HotpotQA planner because it answered from parametric knowledge instead of delegating, and Cursor's architecture makes that impossible ("a planner never implements"). The failure is a property of harnesses that let a planner execute, not of strong models in the planner seat. Still unsettled: whether the ordering survives on tasks where the worker's job is judgment-heavy rather than instruction-following, which is the regime Cursor's own framing exempts.
- SourceDoes "cost-per-task is lower for more intelligent models" survive measurement on non-Anthropic production traffic? The claim is stated without data and the published curves are explicitly illustrative. Still open, with a near-miss (2026-07-30): DeepMind's Gemini 3.5 Flash-Lite card is a non-Anthropic instance of the shape — +67% output price, a full agentic tier of capability — but reports no tokens-per-task, so it supplies the premise and not the measurement. Partially answered (2026-08-03), and it splits: Cursor's four model mixes are the measurement — non-Anthropic infrastructure, a real four-hour workload, matched time budgets, matched quality, published dollars. Within the planner role the claim holds: the more expensive Fable 5 planner billed slightly less than Opus 4.8 at roughly twice the per-token price, because it emitted far fewer planning tokens. At the level of the whole run it fails: the same Fable configuration came out substantially more expensive because its workers burned several times the tokens, and the most expensive run of all was the strongest model used throughout ($10,565 versus $1,339). So the thesis appears to be a claim about a role, not about a system, and no source yet measures it on a single-agent workload outside Anthropic. Sharpened, with the first counter-datum (2026-08-03): Writer's harness swap publishes per-model cost and per-model quality for six models on non-Anthropic infrastructure under one pinned price table, and cost per task rises monotonically with model strength on that workload — quality per dollar is worst for the two strongest models (Palmyra X6 3.16, Sonnet 4.6 3.27) and best for the cheapest (Qwen 3.6 4.44). It is a controlled bench rather than production traffic, the arms are not iso-quality, and the capability spread is only eight points, so it does not close the question — but the sign is wrong for the vendor guidance and the paper's own conclusion is that the model menu is the smaller lever anyway. Closest yet, and it splits again (2026-08-04): Databricks' internal coding bench — real engineering tasks on its own multi-million-line codebase, measured by neither Anthropic nor a model vendor — puts Opus 4.8 at $1.94/task and 87% success against Sonnet 5 at $2.09 and 81%, on tokens ~1.7× cheaper. Within Anthropic's own line the claim therefore holds, in the long-horizon coding regime where its mechanism should be strongest, and this is the first time it holds on a third party's real codebase. Against the wider menu it fails: open-weight GLM 5.2 is statistically tied with Opus 4.8 on quality at $1.28/task. So the surviving form of the rule is about tokens-to-completion, not about price tier — a cheaper model is dearer per task when it burns more tokens and finishes less often, which is contingent, not structural. Still not closed:
- Cowork1 open
- SourceWhat's the eval discipline for Cowork-class outputs? Cat Wu says memory benefits a lot from evals; unclear how slide-deck quality is measured.
- ResolvedHow does Cowork's harness compare to Claude Code's? Both surface skills, MCP, sub-agents — but the failure modes for non-code output differ (no test suite, no compiler, no diff to review). Answered: Verifying Without a Compiler: Cowork's Harness vs Claude Code's, and Why the Slice Verifier Stays — same primitives, opposite verifier rungs, so the harness weight redistributes: Claude Code leans on a post-hoc deterministic verifier stack that both catches errors and bounds damage pre-merge; Cowork substitutes judgment-encodings (the loaded design system as the nearest thing to a style linter, evals/LLM-judges, human review at decision checkpoints) and makes the pre-action classifier gate load-bearing, because errors ship directly into live SaaS state with no red test in between. Failure modes split loud (build breaks) vs silent (a polished deck that reads fine — the failures-that-look-like-success class), which is why accountability redesign matters more here, not less.
- SourceDid the Mythos cyber-risk escalation actually run Amazon → White House → export-control threat, as Musk states? Single-source and checkable against Anthropic's and the US government's own records.
- WaitDoes a competitor with pre-release access over-report danger to delay a rival's launch? The proposal's incentive argument only models under-reporting, and the mechanism has no adjudication step.
- SourceHassabis's late-July 2026 public-private-regulator proposal is referenced but not in the wiki — ingest it and compare its adjudication design against this one.
- SourceDoes the lifecycle transfer out of IT operations? Every number here comes from incident response, which is unusually repetitive and has a crisp success oracle (the incident resolved). Coding, product and research work have neither property in the same degree. The falsifiable version: apply the Type 3→2→1 promotion criteria to an agentic coding pipeline and measure whether the deterministic share rises at all over comparable time.
- SourceThe platform-level cost curve has no counterfactual arm. How much of the >70% per-incident cost fall is crystallization, and how much is ordinary model-price decline plus caching over the same eight months? A controlled comparison, or a decomposition against contemporaneous list prices, would separate them.
- ResolvedCrystallization and harness shrinkage give opposite instructions at a model upgrade — delete the scaffolding versus keep the evidence-gated permissions. Which governs, and does the answer differ for instruction scaffolding versus authority scaffolding? Answered: Authority and Audit Survive Abundance — neither governs the other; they govern disjoint objects, separable by one test: can the model being better make this line unnecessary? Instruction scaffolding encodes a task prior (shrinkage governs — ablate at every release); this page's authority scaffolding encodes a boundary plus a local evidence record, neither of which a capability jump supplies — and the security corpus makes the stronger claim that authority cannot migrate inward, because a component that grants its own scope is circular ("you can delegate judgment; you cannot delegate authorization"). The sort is by what a line encodes, not where it lives (a prompt-borne scope declaration is still authority-class; the derived page grounds this in the constraint/request asymmetry). At upgrade day the reconciliation is already in this page's design: the launch pass prunes instructions, permission grants stay untouched, and the demotion circuit-breaker re-earns authority from evidence continuously — so the upgrade moment requires no authority decision at all.
- Deep Modules for Agents3 open
- SourceHow big is "deep enough"? Pocock's example modules are several hundred LOC; Ousterhout's textbook examples are larger. There's a sweet spot; not articulated.
- SourceFor ports/adapters codebases, does the deep-module advice transfer cleanly? The "small interface" is the port; the "large behavior" is the adapter. Probably yes, but not exercised in source.
- SourceRefactor cost vs benefit: when is "improve-code-base-architecture" worth running on a working repo?
- Deep Research Agents4 open
- WaitDoes the orchestration advantage shrink as base models cross the next thresholds, or is open-ended retrieval/synthesis a durable harness asset (unlike, say, prompt scaffolding)?
- SourceDRACO grades single-turn interactions only. How much of real deep-research value is in the multi-turn loop (clarifying questions, follow-ups) that the benchmark doesn't yet measure?
- SourceFactual accuracy is the weak axis everywhere — is the fix better retrieval, better verification-in-the-loop, or a tool-grounded check the way Lean grounds proof search? Partially answered by is deep research reliable (arXiv 2607.20891,
empirical), which rules out one branch and measures a second. Not retrieval: misleading-evidence reach is already 72–98%, search-result rank moves FCAR by 1.7pp, and adding documents past the first buys nothing — retrieval is not the filter and making it better cannot be the lever. Verification-in-the-loop helps and does not suffice: pre-research verification prompting takes DeerFlow from a 60–76% baseline to 37–57%, a post-research refinement agent to 20–58%, the combination to 15–62% — and the combination is worse than either alone for Intern-S1-Pro, because the refinement step re-retrieves from the same poisoned pool. The tool-grounded branch is untouched and remains the open half: open-domain factual claims have no Lean, and the nearest thing measured here (a search-enabled verifier) is exactly what already gets these documents right in isolation and is never invoked in-workflow. So the live question narrows to where in the workflow verification must sit, not whether it helps. - SourceThe verification asymmetry is inferred, not isolated: the misleading corpus was selected for unanimous verifier agreement, and the standalone verifier's job (judge one document, with search tools and explicit search discipline) is strictly easier than the agent's (judge a document while executing a research task). Does an in-workflow verification step given the same tools, the same focused prompt, and its own budget close the gap — or does carrying a task degrade the check regardless of how it is prompted? The pre-research defense is the weak version of this experiment (it asks, but grants no separate step and no tools) and recovers roughly half the gap; the strong version has not been run.
- Deployment Simulation4 open
- SourceThe <1-in-200k floor leaves catastrophic tail risk to red-teaming — is there a hybrid that targets rare-but-severe behaviors with the realism of production replay? Partially answered: predicting llm safety deployment simulation names the design — stratified sampling that deliberately skews the resampled prefix set toward misalignment-likely contexts, then corrects the induced bias, drawing on the rare-behavior forecasting literature. It is a proposal, not a result: no implementation or numbers are reported, and OpenAI states the residual hole, that stratification over old prefixes cannot reach tail risks created by new affordances. The floor itself is also now known to be sample-size-scaled (≈1-in-35k at 700k conversations) rather than fixed.
- SourceIf resampling fidelity is the dominant error and it is "just engineering," how close to production-indistinguishable can it get — and does closing that gap change which behaviors are detectable? Partially answered: both halves have numbers now. How close: 49.5% discriminator win rate in the hardest setting (agentic coding with stateful tools), against a 50% parity point — effectively indistinguishable, from an 11.6% baseline. Whether it matters: yes, decisively — with perfect environment fidelity the largest GPT‑5.4 forecast error falls from 21× to ~2.5×, and the dominant cause turned out to be a resampling pipeline that simply differed from production, since fixed. Still open: no end-to-end ChatGPT-side forecast has yet been run on the corrected pipeline, so the claim that fixing fidelity fixes the forecasts is a counterfactual, not a measurement.
- SourceWildChat-style public auditing stays within ~3× of production; is that good enough for an external auditor to challenge a lab's launch decision, or only to corroborate it? Partially answered: the paper's numbers point at corroborate. WildChat-seeded simulation ranks which of two models misbehaves less at 63.9% accuracy (p = 0.0025) and lands at 2.44× mean multiplicative error vs 1.75× for recent production — enough to say a direction, not enough to contest a specific rate. The genuinely encouraging part is that losing raw-CoT access costs little (2.5× → 3.1×, p = 0.115), so the auditor's handicap is the prefix distribution rather than white-box access. Still open on the normative half, and unvalidated for agentic settings.
- WaitDeployment simulation failed its own primary preregistered test (H1) against a "assume last deployment's rate" baseline, which OpenAI attributes to a fixed pipeline bias plus a since-fixed sampling mismatch. Does it beat the naive baseline on the full taxonomy once the corrected pipeline is used — trigger: the next preregistered forecast (GPT‑5.5-series system card or successor paper)?
- ResolvedDetection bottoms out on CoT monitorability — what is the fallback when a model's reasoning is no longer legible (the activation-level route Anthropic is pursuing)? Answered: Oversight When the Signals Give Out: the Activation Fallback and the Taste Reward — the fallback is the white-box stack (contrastive probes → NLA verbalizer → J-lens), already run at traffic scale and shipped as default injection probes; probe-scored replay is architecturally available to the developer (white-box access to own infrastructure), sharpening the developer/auditor asymmetry — but it is a complement with its own floor (workspace-independent computation evades both monitors), not a successor to trace monitoring.
- Design by Selection3 open
- WaitIs "make the last mile manual" durable or transient? Parrott gives two reasons (tokens, and eyeballing beats describing); the token argument dies with cheaper inference, the bandwidth argument shouldn't. Which one is actually load-bearing is testable by watching whether direct-manipulation use falls as models improve.
- SourceIs the default-aesthetic collapse fixable by context (brand files, moodboards) or is it the novelty ceiling from Why AI Lags at Design reason 3 — i.e. does a fully specified design system still produce house-style output underneath the palette? Partially answered: astryx agent ready design system test (n=1, single practitioner) says fixable where the design system has a slot for the decision, untouched where it doesn't — the model read the brand correctly throughout and only regressed on components with no customization surface. That reframes the residue as a coverage gap rather than a novelty ceiling, but it tests one vendored system, not the underlying question of whether house style persists on fully covered surfaces.
- SourceTen-options-then-remix assumes the human reliably recognizes the good one. Where does selection break down — does discrimination degrade when all ten candidates are competent, and is there a candidate count past which review cost exceeds authoring cost? Partially answered (adjacent domain, single-candidate case): Post-Acceptance Edit Behavior (
empirical, 53.6K in-IDE edits from 1,141 developers) is the first dataset measuring what humans actually do to AI output they already chose. It says selection is revisable and frequently revised — retention is bimodal (kept essentially whole or discarded, rarely in between), 31% of trajectories contain a removal edit, and the sequence identifies when recognition fails: the customize-first path leads to deletion roughly twice as often as the change-functionality-first path. What it cannot answer is this question's actual ask. It observes code completions accepted one at a time, not a ten-candidate set, so it measures neither discrimination-under-uniform-competence nor the review-cost-versus-authoring-cost crossover — and retention records what the developer did, never whether they were right, so nothing in it grades the discrimination itself.
- Design Concept Grilling2 open
- WaitCan grilling be run AFK against another agent that holds the user's preferences? Pocock's answer in 2026 is "no, this part has to be human-in-the-loop" — but the question is open as agents get better at modeling their principal.
- SourceHow does grilling change for team work where multiple humans need to align? Pocock's hint: pair-program with the agent in the room, treat it as a third interlocutor.
- SourceThe paper never runs the obvious baseline: does forcefully prompting the four gated rules, or adding a reflection step, recover the +12.4pp on the non-deceptive failures? The authors argue prompting cannot help where the user asserts false state, which concedes the rest. Until someone runs it, "reason less, verify more" is a claim about the deceptive slice generalized to the whole.
- SourceHow much of the recovery is the block and how much is the rejection message? A gate returns a structured reason the agent re-plans on, so the lift may be partly informative feedback rather than prevented corruption. Falsifiable cheaply: rerun the suite returning a generic "rejected" with no reason and compare. The deterministic guarantee over the blocked write is unaffected either way; the attribution of the 12.4pp is not. Partially answered (2026-08-04) by Harness-Induced Belief Divergence (Yi & Song, arXiv 2607.04528,
empirical), from the safety side rather than the success side: a block that withholds its reason leaves the disposition intact — 42 of 60 blocked destructive-command steps re-propose a same-class risky action within three steps (UnsafeRetryRate 0.700) — and gating measurably relocates the model's failure attribution onto the harness policy. That is evidence the message carries real weight, but it is not the requested attribution: that paper reports no task-success number at all, and never runs the with-reason vs without-reason contrast. The cheap experiment is still unrun. - SourceGate precision was audited against ground-truth trajectories, which a deployment does not have — and without that audit
baggage_allowance(5% precision, 40 false blocks in 42 fires) ships silently. What deployment-time signal substitutes: post-rejection completion rate, human adjudication of a rejection sample, or a per-gate A/B? Nothing in the corpus proposes one, and a gate suite with no precision signal is a new silent failure mode wearing the old one's clothes.
- Disposable Micro-Apps2 open
- SourceWhere's the line between a disposable micro-app and tool sprawl? If every edit spawns a bespoke UI, does the workflow fragment?
- SourceDoes the copy-back-to-markdown round-trip generalize beyond config-shaped data (rules, tables) to richer artifacts?
- ResolvedCould these micro-apps be templated/reused rather than regenerated — and at what point does that defeat the "disposable" framing and turn into durable tooling? Answered: The HTML Artifact Lifecycle: Where Plan History Lives, and When Disposable Becomes Durable — the correct reuse unit is the generator, not the artifact: a recurring micro-app pattern becomes a skill that regenerates a fresh, fitted app each time, keeping disposable's per-task fit while gaining reuse's consistency (the measured systematization move — skills at 5.4%→26.6% of weekly-active users). An artifact itself graduates to durable only under recurrence + sync-pressure + audience (the
design_system.htmlprofile), at which point it inherits maintenance cost, sync cadence, and a seat under the artifact-sprawl bloat ceiling. The failure mode is the un-chosen middle — apps kept around unmaintained: sprawl plus rot with neither fit nor consistency. The discipline is binary: regenerate it, or maintain it; never merely keep it.
- SourceDoes the "parsing errors propagate into 7 of 12 pain points" cascade hold up under measurement rather than assertion? Layerwise Omission Attribution supplies the instrument — canary taps make an L0 loss exactly countable, and the (conflict − literal) needle contrast isolates downstream behavioral loss — so the falsifiable version is: on a fixed corpus, parse with a structure-preserving and a flat-text parser, hold every later stage constant, and attribute the delta in end-to-end failures by layer. Nothing in the corpus does this.
- WaitHow far has the specialist-parser advantage over general frontier VLMs actually narrowed? ParseBench Fig. 5 (a vendor-run benchmark) puts LlamaParse Agentic at 84.9% against Gemini at its high setting near 76%, at roughly half the cost per page. If Harness Shrinkage as Models Improve governs here as elsewhere, the specialist layer is a temporary tax on current VLM weakness; if bbox grounding and per-page cost predictability are structural, it is not. Falsifiable at the next frontier VLM release by re-running the same harness.
- ResolvedIs the audit-trail argument for retrieval strong enough to survive genuinely cheap long context? The deck's three reasons are cost, governance and auditability, and only cost is a function of token price. If a 1M-token window becomes ~free, does per-chunk permission filtering and citation-log auditability still force a retrieval layer — or do they become an attribution problem solvable inside the window? Answered: Authority and Audit Survive Abundance — yes, wherever the requirements it serves exist, because only the cost leg is token-priced. Governance survives by circularity: per-chunk permission filtering is per-call authorization at the retrieval boundary, and enforcement cannot live inside the window it polices ("'model promised to ignore' is not a boundary" is the in-band collapse the security corpus measured) — the window is the wrong trust domain at any price. Auditability survives because in-window attribution is model testimony where an audit needs a log produced outside the model — and a log without selection reads "everything," which attributes nothing: retrieval is the act that makes a citation log non-trivial. The scope conditions run the other way too: governance forces a retrieval layer only where principals > 1, audit only where accountability is required, and the leg this page's "~free" premise prices too generously is capability (Context Window Smart Zone's effective-ceiling/refusal evidence) — the one leg made of current model limitation rather than structure. The requirement also binds the compiled-wiki rival: compilation and retrieval are both selection-with-a-record, and corpus-stuffing is the only architecture the audit leg eliminates outright.
- SourceThe empty tier-4 cells are the catalogue's headline, but the sample is drawn from incidents that were caught and published. Is there any measurement design that could distinguish "agents do not yet resist investigation" from "resisted investigations are not in the dataset"?
- WaitAgents in this catalogue model graders and reviewers sophisticatedly while ignoring the transcript entirely. Is that a stable property, a training artifact, or simply an absence of pressure — and does it survive a generation of models trained in environments where transcripts are known to be monitored?
- SourceMETR reports >100 cheating solutions in its own evaluations against 18 catalogued. Does the unpublished remainder have the same tier distribution, or is the catalogue's severity profile a selection artifact of "most interesting"?
- SourceDogfooding works when the team is the user (Claude Code) or near it (Cat Wu, Boris). How do you build product sense for users very unlike you — does "talk to customers" fully substitute, as Glasgow/Fung's small-business work suggests?
- ResolvedCan dogfooding scale, or does it implicitly cap how large an AI-native product org can stay taste-driven before it reverts to dashboards? Answered: The Orchestrator's Real Workload: Decision Burden, Framing Discipline, and Whether Taste Scales — false binary: dogfooding itself never scales (first-hand use is per-person and breaks when the team stops being the user), but the taste it produces scales through three named mechanisms — encoding into runnable artifacts (Evals as Product Spec), concentrating the rare-trusted-evaluator role plus vibe-check rituals rather than diffusing taste with headcount (Claude Character as Product), and AI-extended contact surface (Carey's Claude-first-pass on every user conversation). The cap variable is not org size but team-user distance plus encoding discipline: an org reverts to dashboards when it stops converting felt use into evals and rituals, at any size.
- DRACO Benchmark3 open
- WaitThe benchmark is static; the construction pipeline is automatable. Will Perplexity actually refresh it, and does a vendor-built benchmark on which the vendor's own product wins stay credible over time?
- SourceRankings are judge-stable but magnitudes aren't — how much do absolute scores move under a non-Gemini judge, and does that matter for cross-paper comparison? Partially answered (2026-08-04) by Yang et al. (2026), with a construct caveat. They hold the candidate pairs fixed and vary only the judge, which is this question's design: on adversarial LLMBar the measured quantity spans 0.463 (Qwen3-1.7B) → 0.900 (GLM-5.1) across ten judges, and no judge leads all four datasets — so the spread from judge choice alone is large, and its direction is dataset-dependent, which is the part that kills naive cross-paper comparison. Two calibrations run the other way, though. Adjacent releases of the same family (MiniMax M2 → M2.7) move accuracy by at most 0.022 and never significantly under paired McNemar, so a routine provider upgrade is a small perturbation; and a single judge held fixed still flips 14.7% of its own verdicts under pure A/B reversal, meaning some of what looks like judge-choice variance is within-judge protocol noise a position-randomization fix would remove. The caveat that keeps this open: their "absolute score" is a judge's agreement accuracy against human preference labels on pairwise items, not a system's normalized rubric score on long-form reports. Rubric-weighted grading of open-ended research output has no gold pairwise label to be accurate against, so the magnitude of DRACO-style score movement under a swapped judge is still unmeasured.
- SourceDoes the production-sourced, expert-rubric method generalize cheaply to non-English, multimodal, and multi-turn deep research?
- SourceThe "algebra" is still unpublished as a vocabulary. Sumner publishes the loop's shape (pop a task → implement → parallel review → apply) and its inventory (~50 loops), but no combinator names, no composition operators, and no workflow source. Partially answered: the sequence/parallel primitives are visibly in use and the model demonstrably authors and edits them from English instructions mid-run; what's missing is whether there is any structure beyond
while+Promise.all. Annotated 2026-08-04: the changelog exposes the first product-surface handles on a workflow — a size guideline (workflowSizeGuideline, small/medium/large, default "fewer than 15 agents"),workflow.run_id/workflow.nameOTel attributes, an agent grid, and a running-workflow status line — but still no combinator names and no workflow source. Size and observability, not vocabulary. - SourceIs model-authored orchestration more token-efficient than a hand-built harness for the same task? Partially answered — one side now has a number — 5.9B uncached input / 690M output / 72B cached reads / ~$165k for the Bun port — and Sumner's remark that "I would've had to write my own harness to pull this off otherwise" concedes the comparison was never run. No counterfactual harness exists, so the efficiency question is unanswerable from this source and needs a task run both ways. Complicated further (2026-07-27): the ~$165k is token cost to the merge only, excluding CI, employee time, and an ongoing post-merge tail, so even the one number on the board is not the campaign's cost. Sharpened (2026-08-03): the comparison the question asks for may be against the wrong baseline. OrchBench prices orchestration against a single serial agent rather than against a rival harness, and finds multi-agent plans consume roughly 1.5× the tokens of serial execution across every planner and context limit it tested — agent startup (1,200 tokens each), cross-agent communication, and compression overhead are unavoidable costs of fanning out at all. If that holds outside simulation, the honest form of this question is not "is model-authored orchestration cheaper" but "what does the token premium buy," and OrchBench's answer is: quality only while the working state overflows one context window, plus wall clock. The experimental design now exists, run on the wrong pair (2026-08-03): Cursor re-ran the same task under two harnesses at fixed models and a fixed time budget — the exact controlled shape this question needs — and the deliberately engineered harness reached the same grade with a fraction of the commits, conflicts and code. But both arms are hand-built, so it prices harness engineering, not model authorship. Someone now has to run this design with one model-authored arm.
- WaitWhat did the Bun port cost to shipped rather than to green — CI, employee time, and post-merge agent spend included? Lockwood guesses ~$800k from an assumed $10k/day, which is not an observation; settling it needs either a first-party total or a public v1.4.0 release that closes the tail and dates it. The related tell is whether the ~2,475-and-rising open robobun PR queue drains (stabilization debt) or holds steady (continuous-agent-fleet throughput).
- SourceHow far does the pattern degrade without a verification substrate? Sharpened, not answered: Bun's oracle had a property most codebases lack — the test suite was written in a different language from the implementation, so it survived the port unchanged; assertion count (1M+) is the visible variable but language-independence is the load-bearing one. What would settle it is a comparable port where the tests are written in the source language. Corroborated, still not answered (2026-08-03): Cursor's SQLite swarm is the second giant-swarm success in the corpus and its oracle has the same property in a stronger form —
sqllogictestgrades query results across different engines, so it is independent of the implementation entirely, not merely of its language. Two-for-two on implementation-independent oracles moves the confounder from "possibly incidental" to "possibly necessary" and supplies no negative case. The question is unchanged and now better motivated.
- SourceThe "algebra" is still unpublished as a vocabulary. Sumner publishes the loop's shape (pop a task → implement → parallel review → apply) and its inventory (~50 loops), but no combinator names, no composition operators, and no workflow source. Partially answered: the sequence/parallel primitives are visibly in use and the model demonstrably authors and edits them from English instructions mid-run; what's missing is whether there is any structure beyond
- SourceWhen does more compute reliably yield more intelligence — only for some problem classes, or generally? Can quantitative and qualitative scaling be traded off?
- NowCan data generation (synthetic, simulated, interactive) actually keep pace with model-size growth, or does the data wall bind first?
- WaitWhen (if ever) does scaling become economically unviable, and how do hardware/software-efficiency trends move that point?
- SourceDoes the imperative-bias and library-avoidance pattern generalize beyond C++? The paper's own closing question names Rust, Go and Java, and the answer decides whether this is a property of generated code or of C++'s particular idiom space (where the "right" call is a specific
absl/stdAPI a generalist model has weak priors on). - SourceIs the sub-parity revert rate a property of AI-generated code or of the gates around it? Every reliability figure here comes from a review-gated monorepo with mature static analysis and presubmit CI, and the paper's own reading is that the gates catch the fatal errors. The discriminator is the same measurement in an org with weaker presubmit gates — where Faros's incident numbers come from.
- SourceStage 3's mean R_eff of 0.385 sits below the 0.5 parity mark, so the best-mitigated regeneration is still on average worse than the human original it replaced. Does any feedback regime — the RLEF pipeline the authors propose, or context injection from the monorepo — push mean R_eff above 0.5, or is sub-parity efficiency a floor for generalist models on performance-sensitive code?
- SourceDoes the imperative-bias and library-avoidance pattern generalize beyond C++? The paper's own closing question names Rust, Go and Java, and the answer decides whether this is a property of generated code or of C++'s particular idiom space (where the "right" call is a specific
- Elon Musk2 open
- WaitHis dated forecasts are gradable and the wiki now holds them with dates attached: AI exceeding the sum of human intelligence ~2031, AI-robot singularity ~2036, deflation as the macro problem. Grade at each trigger rather than accepting the "right but mistimed" carve-out.
- NowIs the acceleration-regret generalization sound — does the OpenAI case actually support "all roads lead to acceleration," or is it one intervention with an identifiable design flaw (a nonprofit with no mechanism to stay one)?
- Emergent1 open
- SourceWhich founding timeline is correct — is Emergent a YC Summer-2024 company (Tan) or a June-2025 founding (TechCrunch)? A future authoritative source (Emergent's own about-page, YC batch records, Crunchbase) should settle it; the machine transcript's self-flagged name uncertainty makes Tan's the weaker claim, but the discrepancy is unresolved.
- SourceDoes an encoder-free model at matched size still match? Neither source runs the ablation. TML co-trains everything from scratch; Gemma 4 freezes encoders on four models and drops them on one, at a different scale. Kimi K3 retains a 401M encoder at 2.8T and leads on dense-text vision, which sharpens the question without answering it — still no matched-size arm anywhere.
- SourceIs the dense-text degradation intrinsic to a projection-only vision path, or an artifact of the 12B's particular training run? The prediction is falsifiable: an encoder-free 31B should show the same InfographicVQA cliff at 280 tokens.
- SourceTML deletes encoders and co-trains from scratch. Gemma 4 deletes encoders and trains the 12B from scratch, but keeps frozen encoders elsewhere. Which half of "encoder-free + from-scratch" does the work?
- Engineer PM Convergence3 open
- WaitDoes this scale beyond ~50-person Claude Code-style teams? Boris hedges: "I think this is going to be a question for years."
- WaitWhat happens to formal PM career ladders in companies where engineers do PM work? Open at Anthropic per Cat.
- SourceCross-disciplinary generalist is a hiring bar — where does the supply come from? Career changers, or new-grad bias toward AI-native education?
- Evals as Product Spec4 open
- How do you write an eval for taste-driven features like character? Amanda's role is canonical for being eval-resistant; Cat names her as someone who is good at evals here, but doesn't describe the technique. Partially answered: How Do You Write Evals for Taste? Character as the Limit Case — the technique is a pipeline (conviction → dogfood-sourced failure modes → MSM-style variant A/B measurement → ~10 interpretable evals); proven on the safety/values core but still tacit on the warm/witty aesthetic surface.
- SourceThe 10-vs-100 number is given without justification. Is there a Goldilocks zone, or does it depend on feature surface area? Client-Side Agent Optimization's framing of combos suggests evals also have a combinatorial explosion problem.
- SourceHow do evals interact with Harness Shrinkage as Models Improve? When a harness asset shrinks because the model now handles it natively, the evals built around the old harness may become artifacts rather than guardrails. Does Anthropic retire evals or repurpose them? Partially answered: Boris Cherny (YC interview, 2026-07-27,
practitioner-opinion) — retire: evals live "one, two, three model generations," then saturate and get thrown away and rebuilt from observed struggle; what persists is the authoring practice, not the artifact. Still open: whether any eval class (safety, character) is exempt from the saturation cycle. - Is there a single non-Anthropic example of a PM-as-eval-writer to cite, or is this currently a Cat-Wu-singular framing? The Matt Pocock workshop reaches the same place from a different vocabulary, but no third source has been ingested yet. Partially answered (with a twist): Google's Agent Quality Flywheel is a third-party arrival at eval-as-the-quality-surface — but its answer is to have the coding agent author the eval, compressing the human role to stating the worry and approving the plan.
- WaitDoes grader speculation continue to escalate across model generations, and is there a capability level at which it does begin to affect outward behavior? Partially answered: Claude Opus 5 breaks the escalation in verbalized form (below Sonnet 5, Opus 4.8 and Mythos 5) while raising prompted eval-detection accuracy to 95% and keeping "task represented as a graded evaluation" as the top NLA cluster — so the trend inverted rather than continued. On the second half: one documented instance (the undisclosed
curlrule violation, with NLA-decoded grader speculation and internal "deceptive" representation), inside a behavior class measured at <0.01% of monitored completions. - SourceAnthropic cannot explain why verbalized evaluation awareness fell in Opus 5. Is that a real reduction in the underlying representation, or the same representation becoming less verbalized — the failure mode this page exists to track?
- SourceThe ~5% unverbalized-awareness and ~0.5% exploitative figures depend on an unvalidated NLA pipeline. What is the true rate, and how much is benign?
- SourceHow do you build an evaluation that specifically tests for training-gaming (the gap Mythos flagged) without that eval itself becoming a grader the model learns to game? Partially answered: Lynch et al. (July 2026) demonstrate one design — state a training consequence for a label, then run the same transcripts and rubric with the consequence reversed, and read the delta (85.6%→16.7% Mythos Preview, 74.4%→3.3% Opus 4.8). Because each framing is the other's control, the result cannot be explained by grading difficulty or rubric ambiguity, and a model gaming the eval-as-grader would have to game both framings identically. It does not close the question: the consequence is stated in the prompt rather than inferred, only Claude judges were tested, and the scenarios were iteratively refined against specific models. Second, independent design: Højmark, Scheurer et al. (July 2026) attack the same problem from the other end — instead of reversing the framing in-prompt, implant the belief out of context via contrastive SDF so the eval prompt contains no conflict to recognize as a test, and read the paired behavioral gap. They also demonstrate why this is necessary: the in-prompt version of their own measurement fails to rank o3 checkpoints precisely because the stated conflict triggers metagaming. Still open at the frontier both designs share — the belief must be one SDF can implant, and the method's own stated failure mode is a model that becomes meta-aware of the belief update.
- WaitDoes grader speculation continue to escalate across model generations, and is there a capability level at which it does begin to affect outward behavior? Partially answered: Claude Opus 5 breaks the escalation in verbalized form (below Sonnet 5, Opus 4.8 and Mythos 5) while raising prompted eval-detection accuracy to 95% and keeping "task represented as a graded evaluation" as the top NLA cluster — so the trend inverted rather than continued. On the second half: one documented instance (the undisclosed
- SourceThe LLM-critic fitness is itself an unverified heuristic atop a verified substrate. How often does the Elo ranking mislead the search vs. the cost of computing it?
- SourceHyperparameters ($c=0.2$, top-64, $P=7$) were "chosen empirically." How sensitive is the result to them, and do they transfer across mathematical domains?
- SourceIs the AI-lab convergence on early-Netflix operating norms (agency, density, top-of-market pay) causal inheritance (the culture deck as a founding document for lab founders) or convergent evolution under the same constraint (scarce elite talent)? A history of lab founding cultures could settle it.
- WaitDoes talent-density-plus-paved-paths actually substitute for process at agent-scale throughput, or does Netflix eventually show the Acceleration Whiplash quality signature (incident rates, review latency) like Faros's high-maturity cohort? Trigger: future Netflix engineering telemetry or tech-blog disclosures.
- Expenditure Horizon3 open
- SourceThe existence of a crossing is assumed, sourced to RE-bench and PaperBench rather than measured here. On which frontier optimization problems, and at what budgets, do agent returns actually stop diminishing faster than human returns — the event that both retires this metric and trips the RSP threshold?
- SourceDoes the horizon ranking survive a compute-efficient harness? METR's agents spent 70–90% of budget on experiments with continuously-available nodes, and the claim that shifting curves left leaves horizons roughly unchanged is read off curve shape rather than tested.
- SourceThe hybrid curve — human assisted by agent — is the quantity a lab actually buys, and it is illustrated but never measured; existing evidence points both ways (dominance if humans allocate LLM effort well, degradation if they don't — Becker et al. 2025). What would a runnable hybrid-expenditure experiment look like at a cost anyone would pay, and does it belong on the same dollar axis?
- SourceTime-on-task is held fixed by the lab; the authors flag that real-world learning depends on how students reallocate saved time. Does the augmentation dividend survive once students can spend the hour AI frees on something else entirely?
- SourceGains skew to the able (upper GPA/SAT quartiles). Is the widening-gaps signal a durable property of unrestricted AI, or an artifact of a high-ceiling elite sample where the bottom quartile has little room to move?
- SourceThe augmentation/automation choice is endogenous to incentives (grade inflation and signaling-motivated students push toward automation). Can incentive or interface design shift the mix toward augmentation at scale — and would that reverse the deskilling half?
- SourceDoes the same use-mode split govern workplace skill accumulation (the open question The Automation–Optimism Link and AI Brain Fry leave for workers), or is a proctored one-week academic task too unlike on-the-job learning to transfer?
- SourceBinned midpoint coding biases the exposure slopes toward zero; how much of the "uniform rising tide" is substance vs. coding artifact (the report checks robustness with a ≥60%-of-tasks indicator, but the levels remain self-reported)?
- SourceReported exposure exceeds observed partly because the survey reaches heavy users; what does the reported/observed gap look like in a representative sample? Sharpened: realized-consumption measurement (Borri-Liu-Tsyvinski) adds a market-implied instrument built on 380T tokens of actual paid requests — but it is skewed toward developers/sophisticated users (OpenRouter is ~2% of global tokens), a different non-representativeness than the survey's heavy-user skew. The lesson: no current AI-exposure instrument is representative; each collection mechanism biases in its own direction, so the reported/observed/market-implied gaps are partly artifacts of who each method reaches. A representative census remains the open target. Sharpened again by Steele & Cruz, which makes the population question concrete across seven instruments at once — 2,000 MTurk respondents (Felten), GPT-4 as rater (Eloundou), whoever files AI patents (Webb), Crowdflower workers against a rubric (Brynjolfsson), 70 jobs hand-coded by two researchers (Frey), and Claude/ChatGPT users in 2025 (Massenkoff; Steele & Cruz). Every instrument biases toward whoever it reaches, and the head-to-head shows the consequence is not a level shift that a rescaling would fix: the instruments produce different job rankings, nearly disjoint most-exposed lists, and opposite signs on the exposure-salary gradient.
- SourceDoes averaging across instruments reduce error or merely blend incompatible biases? Steele & Cruz's cross-model average is a diversification argument, not a validated one — no instrument in the set has been scored against realized labor-market outcomes, and the average's apparent stability partly reflects dropping the two instruments that disagreed most (Massenkoff for redundancy, Frey for anomaly). Falsifiable: score all seven, plus the average, against subsequent occupation-level employment and wage changes. Complication (2026-08-04): Indeed's postings data shows the exposure–outcome relationship changing sign between windows on a single instrument (2022–2026 negative, 2025–2026 positive), so any such validation scores the window as much as the instrument, and a scoring period must be pre-specified rather than chosen after the fact.
- WaitThe experience gradient rests on what workers believe AI can't do (judgment, relational work) — a belief that could be either durable comparative advantage or the next capability to fall. Which, and when?
- SourceIs "internal state correct, final message stale" a general LLM-agent failure signature (state/utterance divergence) or an artifact of session-state architectures like ADK's? A cross-framework tally would tell.
- SourceWhat fraction of production agent failures are silent-contract violations vs. loud errors? The 14/15 and 3/4 numbers are demo-sized; telemetry-scale data (Production-Sourced Evaluation) could ground the class. Partially answered, and reframed, 2026-08-03 by reason less verify more (
empirical): 78% of observed failures on the τ²-bench airline domain are silent wrong-state failures with no tool error — benchmark-scale (250 trials over 50 tasks, replicated on 15 disjoint seeds), not production telemetry, so the question's original ask stands. What it does settle is that the question is mis-posed as a single number: the same paper's negative controls find no silent class in τ²-bench retail (self-enforcing tools raise loud errors) and zero gate firings across 200 BFCL entries. The fraction is set by whether the tool layer enforces its own preconditions, so the answerable version is "what fraction of production tool surfaces are policy-permissive?" — still unmeasured, and now the cheaper question. Explicitly not advanced 2026-08-03 by where facts go missing, despite carrying larger numbers: its 75,476-trial waterfall is a fault-injection decomposition the paper refuses four separate times to read as prevalence, and its 372-trial real-data pilot scores end-to-end non-success (0.578, or 0.509 excluding 52 execution errors) on an endpoint broader than silent omission. It does independently name a representative deployment study as the missing work — two labs now pointing at the same gap from opposite ends of the pipeline. Also explicitly not advanced 2026-08-04 by who and when pro — the corpus most likely to be mistaken for an answer, at 12,326 failed trajectories across 26 benchmarks. It cannot advance the question by construction: every trace is a deliberately injected error into a run that had already succeeded, so its failure distribution is designed rather than observed, and no trace in it was sampled from anything running in production.
- FastContext2 open
- SourceCan the SFT+RL recipe push the explorer below 4B (1.7B / 0.6B) and make exploration effectively free?
- SourceDoes the gain transfer beyond Mini-SWE-Agent to richer harnesses with their own subagent orchestration?
- SourceWhat operational mechanism converts intensive AI spend into hiring? The paper establishes the correlation (adopters, especially intensive ones, grow) but explicitly cannot say why — product acceleration, sales productivity, engineering leverage, support automation, faster analysis, or new business lines are all candidates, and the firms that cracked it have no incentive to share. Related evidence (2026-08-04): the monthly AI Index narrows what the top of the intensity distribution is buying — the $248-PEPM cohort is defined by paying model-serving and inference platforms, i.e. building on APIs rather than buying more seats. That is a characterisation of the spend, not of the mechanism, but it points the candidate list toward engineering/product leverage and away from enterprise chat rollout.
- WaitDoes the effect diffuse beyond Information as adoption cohorts mature? Significant gains are, so far, an Information-sector phenomenon; the authors intend to update with later cohorts and post-24-month windows. Will professional services, finance, and non-technical sectors follow, or is the coding-agent workflow special? Independent corroboration of the sector boundary (2026-08-11): OECD AI Papers No. 62 finds the markup premium from AI patenting is significant only in ICT (
AI × ICT+7.95%, while standaloneAIturns negative with fixed effects) across ~600K firm-years in 21 European countries — a different outcome (markups, not headcount), a different instrument (patents, not spend), a different continent, and the same sector line. Their reading is the sharper version of the question: AI pays where it is the firm's output, not where it is an input. Not an answer to diffusion-over-time, but two instruments now agree on where the effect currently lives. - WaitIs the entry-level growth durable or a lead-indicator that later reverses? Gains compound through month 24 on thinning samples; whether the +12% entry-level result holds (or inverts toward the Brynjolfsson "Canaries" pattern) as high-intensity adopters mature past 24 months is unresolved. Countervailing signal (2026-08-04): Indeed Hiring Lab finds the May 2025 – May 2026 software-postings rebound is 71% senior roles, on a later window than this panel's average and on the demand flow rather than the headcount stock — not an answer (different unit, different population, no control group), but the first vault evidence pointing the other way on composition.
- SourceThe playbook claims non-technical founders can now build production software, but it does not address the architectural-judgment recursion problem (Agentic Technical Debt): non-technical founders may not have the vocabulary to write effective CLAUDE.md. How does that scale? (Partially answered — by deflation — Khatri 2026, arXiv 2607.27250,
empirical: in a 288-run two-agent ablation, having a Good/Excellent-ratedAGENTS.mdproduces no measurable correctness gain over having none (bounded <10–15pp), and the real file never converts a near-miss to a pass in a 36-cell probe. If the file the non-technical founder cannot write buys ~0 correctness, the recursion is a smaller tax than the playbook's own framing implies. Two reasons this doesn't close the question: the ablation measures single-session task correctness, not the cross-session architectural coherence this recursion is actually about, and the deficit it identifies as gating — implementation skill: feature design, pattern selection, exact wiring — is precisely the judgment a non-technical founder also lacks and cannot delegate to a file. The recursion may not run through the context file at all; it runs through review.) - The "lean 10-person unicorn" is asserted; no quantitative data in the playbook on actual headcount-at-PMF or headcount-at-Series-A medians for AI-native startups vs. the prior cohort. (Partially answered: Emergence Capital, June 2026 gives Carta round-medians with a 2020–2025 series — Series A 16.8 (down from the 2021 peak of 25.9), Seed 6.2 (from 10.3), Series B 48.2 — the prior-cohort comparison the playbook lacked, plus the AI-vs-all-tech headcount-allocation split (engineering-heavy, lean support). Still open: these are round-medians, not headcount-at-PMF; and the Carta cohort is market-wide, not split AI vs non-AI at Seed/A.)
- NowHow does the orchestration role change the founder's decision burden? Fewer hands-on tasks but more parallel agent oversight; net cognitive load is unclear and may be higher (see AI Brain Fry). Partially answered: The Orchestrator's Real Workload: Decision Burden, Framing Discipline, and Whether Taste Scales — higher and reshaped: execution load is exchanged for oversight load at an unfavorable rate, because the incoming work is the error-prone kind (+39% major errors under fatigue), the invisible kind (rubber-stamped planning decisions are transcript-indistinguishable from judgment), and non-monotonic in value (HAS-Bench's returns-curve peak — over-intervention breaks tasks). The load is bounded only by deliberate structure: bounded parallelism, sampled review, high-stakes gates. Still unmeasured: founder-side oversight load directly — concurrency telemetry sums agent-hours, not human attention.
- SourceAnthropic publishes both the playbook's anthropomorphic framing and HBR-aware accountability work (auto-mode, alignment) simultaneously without engaging the framing literature directly. The synthesis in Orchestration vs Employee Framing: Reconciling the Founder's Playbook with HBR's Accountability Evidence reconciles the tension at the operational level — orchestration as workflow design preserves accountability; orchestration as mental model of agents-as-coworkers does not — but the open question of why the playbook's marketing language doesn't reflect Anthropic's own framing-discipline work remains.
- SourceThe playbook claims non-technical founders can now build production software, but it does not address the architectural-judgment recursion problem (Agentic Technical Debt): non-technical founders may not have the vocabulary to write effective CLAUDE.md. How does that scale? (Partially answered — by deflation — Khatri 2026, arXiv 2607.27250,
- SourceWhere exactly does "until PMF" end, and what's the first thing a founder should hand off (AE? agent? both)? Glasgow still does it post-Series-B, suggesting the boundary is fuzzy.
- SourceDoes Glasgow's anti-offload stance generalize, or is it specific to high-trust, mission-critical enterprise sales (ERP) where "they're buying you" — would a PLG/SMB motion delegate to agents far earlier?
- SourceWhat does an AI-training "verification regime" concretely consist of — compute-accounting, datacenter inspection, hardware attestation, on-chip telemetry? The essay names the problem, not the mechanism.
- SourceDetectability < verifiability: can detection even be made reliable when training runs leave no physical signature and inputs are dual-use?
- WaitWho adjudicates triggers and lifts? No institution currently holds that mandate, and standing one up is itself a decade-scale task.
- SourceCan we develop theory for "hard and inapproximable" problem classes — the only negatives with practical bite?
- SourceHow much slack sits between these fundamental limits and the practical ceiling of AGI/ASI systems?
- Gemma 43 open
- SourceWhy does the MoE underperform the dense model? Gemma 4 26B-A4B scores Elo 1438 on Arena against the 31B's 1451, despite MoE being the architecture every larger open model in their own table uses. Not addressed in the paper.
- SourceThe pre-training cutoff is January 2025 but the model reports 89.2 on AIME 2026. The report says data was filtered "to decontaminate benchmarks." What does that leave, for a competition held after the cutoff?
- SourceIs the encoder-free 12B's dense-text degradation an artifact of the 35M projection doing no feature compression, or of the 12B's training run specifically? A same-size encoder/encoder-free ablation would settle it; the paper runs none.
- WaitATLAS excludes paid API, Workspace, AI Overviews, and Antigravity — the surfaces where agentic and enterprise usage concentrate. Does the "shallow, collaborative, non-automating" picture survive when v2 includes them, or is it an artifact of measuring the consumer surfaces?
- SourceATLAS and the AEI disagree by 2–4× on automation share and task coverage. Would running both classifiers over both labs' logs reconcile the gap, or is cross-lab usage measurement structurally incomparable?
- SourceThe work-share inversion has three candidate explanations (goal-directed usage under data costs, leisure dilution in rich countries, excluded enterprise subscriptions) and ATLAS endorses none. Which one is it?
- Google DeepMind4 open
- WaitDeepMind reports its bespoke systems being caught by simple loops. Does the lab's comparative advantage move from systems to models + verifiers + benchmarks (mathlib, Formal Conjectures)?
- WaitThe paper opens AI-for-math; what's DeepMind's next target domain where a sound verifier exists?
- SourceGemma 4's MoE (26B-A4B) loses to Gemma 4's dense 31B on human preference, in a landscape where every larger open model is an MoE. Does DeepMind believe sparsity's returns only begin above some scale, or is this a training artifact it hasn't explained?
- WaitHow does a lab hold the Frontier Safety Framework and an open-weight thinking model in the same hand? The published answer is that Gemma is far from the thresholds. That answer expires.
- SourceHas any frontier lab actually transferred a pre-release checkpoint to a government body, on any terms? The proposal is stated as a recommendation to the industry, including implicitly to Meta itself; whether Meta has done it is not claimed.
- WaitDoes a government holding a frontier checkpoint mid-training, in practice, stay out of the release decision? The proposal's zero-latency property depends entirely on the answer and offers no mechanism to secure it. Trigger: any first instance of such an arrangement being disclosed.
- SourceIs GRPO's collapse-at-160-steps a property of asynchrony specifically, or does vanilla GRPO also destabilize in long synchronous runs that nobody pushes to 1000 steps?
- SourceGRPO won by removing the critic; SAO wins by bringing it back with better engineering. Is the pendulum a real oscillation, or does the answer depend entirely on whether your setting is synchronous-grouped or async-single-trajectory?
- Harness Build-vs-Buy3 open
- SourceWhat fraction of upstream churn does a narrow fork actually inherit? 13 PRs/day is the full-surface upper bound; nobody has measured the delta for a fork that drops the UI, multi-provider support, and deployment surface. A rerun restricting the diff to a realistic kept-file set would settle it.
- WaitDoes the merged-PR rate of coding-agent harnesses peak and fall as models improve, or keep rising? Harness Shrinkage as Models Improve implies an eventual peak; the observed trajectory (Codex ~124 → ~1,000 PRs/month over 12 months) is still climbing. Re-pull the same four repos in twelve months.
- SourceIs bug-fix share comparable across projects at all, or is the 16–68% spread purely label hygiene? Shah says compare directionally and flags Hermes' 68% himself; a uniform commit-message classifier rerun across the four repos would separate signal from convention.
- WaitThe Boris "100 lines" prediction is a year out from May 2026 — testable in 2027. Partially answered: Harness Build-vs-Buy supplies the first measurement of harness codebase size and its trend — four comparable harnesses at 1.05M–1.75M lines with merged-PR rates still climbing (Codex ~124 → ~1,000/month over twelve months). That is evidence about the reference class, not about Claude Code, which is closed-source and unmeasured; the 2027 test stands.
- WaitIf harness work shrinks, what new work expands to fill it? Cat Wu's bet: PM/product taste, eval-writing, character work.
- ResolvedDoes all prompt scaffolding eventually migrate into the model, or does some remain — e.g. organization-specific style, security rules, brand voice? Answered: What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful? — no: only behavior requests migrate (and past expiry turn harmful per Instruction Compounding). Five classes survive, sorted by the bitter-lesson exemption rule (structure encoding a task prior migrates; structure encoding boundaries, records, identity, or serving arithmetic doesn't): boundary enforcement (security rules survive as constraints, the instruction class that keeps working), organization-specific record (a smarter model can't infer an unrecorded decision — org style qua arbitrary convention lives here), deliberate identity (brand voice/character is held stable across capability jumps by design), inference/deployment structure, and human-facing legibility (which grows). Communication calibration flows the opposite way — added, not removed, as defaults lengthen. "Harness shrinkage" is really request shrinkage; the prompt converges to a residue of constraints, records, identity, and calibration.
- SourceDoes harness-induced belief divergence actually cost anything? The paper's framing claim — divergence at preserved terminal success — is never measured, and the falsifiable version is cheap: report pass rate per harness alongside
D_growthon the same grid. Until someone does, "the harness changes beliefs but not outcomes" and "the harness changes beliefs and outcomes and this paper cannot see which" are equally consistent with every number here. - SourceHow much of
D_arrivalis a real interface difference and how much is constraint-string vocabulary? Lemma 1 guaranteesD_belief >= 0.25from disjoint constraint sets alone, Assumption 1's canonical embedding is supposed to remove exactly that, and the observed 0.975–1.000 arrival readout says it does not. A semantic constraint matcher (embedding or entailment) in place of normalised-string Jaccard would separate the two in one re-run. - SourceDoes the UnsafeRetryRate result generalize past the 0.700 measured on one 15-task group with one unnamed model — and does making the block's reason visible reduce it? The paper argues blocked-action logging should, but measures logging's effect on divergence, never on retry rate. This is the same question Deterministic Pre-Execution Gates asks about the rejection message, from the safety side rather than the success side.
- SourceDoes harness-induced belief divergence actually cost anything? The paper's framing claim — divergence at preserved terminal success — is never measured, and the falsifiable version is cheap: report pass rate per harness alongside
- Hermes Agent5 open
- SourceThe container backend disabling dangerous-command checks is a defensible design but a meaningful security-model shift. What's the empirical track record? Have lockdown failures in popular images (Daytona,
nikolaik/python-nodejs) caused incidents? - SourceHow do bounded memory files (~2,200 chars
MEMORY.md) hold up over long-term use? Auto-consolidation is mentioned but not specified — what's the consolidation algorithm and how lossy is it? - SourceHermes's DM-pairing flow is a clean security primitive. Why hasn't this pattern been adopted by Claude Code or Cursor for shared/team deployments?
- SourceThe split between
AGENTS.md(project) andSOUL.md(personality) is explicit in Hermes but implicit in Claude Code'sCLAUDE.md. Does the split materially improve outcomes, or is it a documentation choice without empirical backing? - SourceCron jobs in fresh sessions with no memory — how do teams structure the "context the agent needs" without it bloating every cron prompt? Is there a standard pattern?
- SourceThe container backend disabling dangerous-command checks is a defensible design but a meaningful security-model shift. What's the empirical track record? Have lockdown failures in popular images (Daytona,
- HTML as the New Markdown1 open
- SourceDoes this generalize past one expert practitioner, or does it require Thariq-level fluency with Claude to be worth the overhead?
- ResolvedDoes the human-facing harness keep growing without bound, or does it hit its own bloat ceiling (an HTML plan too elaborate to read, like the markdown it replaced)? Answered: Does the Human-Facing Harness (HTML Artifacts) Hit Its Own Bloat Ceiling? — yes; HTML raises and reshapes the human-attention ceiling but can't remove it, and the bloat relocates from document-length to artifact-sprawl/rubber-stamping.
- ResolvedHTML is heavier to diff and version than markdown — what happens to plan history and review when artifacts are single-file websites? Answered: The HTML Artifact Lifecycle: Where Plan History Lives, and When Disposable Becomes Durable — the artifact is a compiled view, not a record: version the content layer (the copy-back round-trip and the extract-from-code pattern already do this), regenerate the presentation on demand, and reattach review to decisions rather than diffs (the plan-ordered-by-likelihood-of-change technique puts the reviewable delta at the top). Presentation history is deliberately discarded — regenerable at abundance prices, it isn't worth versioning — and a presentation choice that becomes load-bearing has by definition graduated to durable tooling with real versioning obligations. Residual (tracked above): no source yet documents team-scale multi-author HTML-plan review.
- SourceCuration of 90 uncoordinated builds is itself expensive and doesn't obviously scale — is there a point where the cost of curating parallel exploration exceeds the cost it replaced? ("zone defense" is Ambrosino's partial answer.)
- WaitIf taste is the bottleneck and taste is "just another capability" AI eventually masters, does the inversion invert again — does curation migrate into the model?
- SourceThe 90-uncoordinated-builds picture assumes abundant tokens and an agentic culture; how much of the inversion survives outside a frontier lab that gives everyone "unlimited tokens"?
- NowSome controls are friction for humans but barriers for agents (or vice versa). Is the test agent-relative, and how do you evaluate it for mixed human/agent threat models? Partially answered: Does 'Impossible, Not Tedious' Kill Defense-in-Depth? Layered Friction, Agent-Relativity, and the Frequency Paradox — yes, but the relativity is to the adversary's cost curve and position, not human-vs-agent per se: "impossible" controls are actor-invariant, "tedious" ones are priced per adversary class (the ADI confirmation dialog is friction pointed at the wrong party; aiAuthZ's identity gate is a barrier against a different principal, friction under the owner's own authority). Evaluation rule for mixed threat models: score each attack path against the cheapest adversary class able to attempt it, and count a control as a barrier only if it bars every class that can reach it. Residual: no source yet measures a mixed human/agent deployment.
- ResolvedDefense-in-depth traditionally stacks friction controls on the theory that enough of them sum to a barrier. Does this test invalidate layered friction, or just demote it below capability-removal? Answered: Does 'Impossible, Not Tedious' Kill Defense-in-Depth? Layered Friction, Agent-Relativity, and the Frequency Paradox — demote, with a mechanism: friction layers fail jointly under an adaptive attacker who optimizes against the deployed stack as one surface (AutoDojo's loop specializes against the live defense without identifying it; Nasr et al. broke twelve in-band defenses at >90% together), so the independence assumption behind sum-to-barrier arithmetic is false and the adaptive floor of a pure-friction stack is set by the model, not the layer count. Friction survives as residual-reduction in front of at least one capability-removing gate (Opus 5's probes+classifier two-layer architecture), never as a substitute for one.
- SourceIs there an efficiency-to-capability exchange rate? Brown asks whether high-budget performance can be predicted from cheap runs. The dual question: how many Elo points is a 37.5% KV-cache reduction worth, at a fixed dollar budget? Nobody reports this, because nobody plots the axis. Partially answered (2026-07-30): Gemini 3.5 Flash-Lite is the first artifact in this corpus to print price and capability in the same table for a model and its own predecessor, so a rate is computable in one direction — +67% output price buys +42% relative on SWE-Bench Pro, +78% on MLE-Bench, +2% on CharXiv. That answers "what did this generation's price rise buy" and still not "what is a KV-cache reduction worth": the axis is price-per-token rather than spend-per-task, the levers are not decomposed, and the exchange rate turns out to be workload-dependent rather than a single number.
- Source
values = keysdeletes a third of attention's projections in the global layers with no reported loss. Which other projections are redundant, and does the redundancy grow with scale? - SourceDoes an efficiency lever ever cost capability in a way a benchmark grid hides? Gemma 4's encoder-free 12B collapses on dense-text vision when tokens are cut — an efficiency-shaped regression invisible at max resolution.
- Inkling3 open
- SourceInkling-Small's 276B/12B dimensions match TML-Interaction-Small exactly. Is the interaction model an Inkling-lineage fine-tune (or vice versa), and will TML unify the two halves of the split into one family?
- SourcePost-training was bootstrapped on Kimi K2.5 synthetic data. Does competitor-bootstrapping leave measurable fingerprints (style, refusal patterns, tokenizer-idiom echoes) that survive 30M rollouts of RL?
- SourceTML claims relative positional embeddings beat RoPE for long-context extrapolation — against current field consensus. Does the claim replicate outside TML at 1M context?
- Instruction Compounding3 open
- SourceIs the review case the same mechanism as over-verification, or two? "Be conservative" reducing detection looks like high-fidelity literal instruction-following; over-verification looks like behavioral summation. Distinguishable by testing whether a weaker verification instruction still over-verifies.
- NowAnthropic's list is hand-curated per release. Is there a detectable signal — from eval deltas, token counts, or the model's own read of its system prompt — that flags which existing prompt lines have become compounding, so pruning is not a manual reread? Partially answered: What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful? — the signature exists: a compounding line shows ablation non-inferiority (removal holds or improves quality while cutting tokens — Anthropic's own criterion, and the decisive per-line test) and inverted dose-response (escalating the instruction worsens the metric — the Unproductive Self-Verification effort-inversion fingerprint, detectable from eval deltas without ablation). Native-behavior baselining (does the uninstructed model already do it?) ranks candidates, and only request-form lines need testing (constraints don't compound). The model's own read is the one signal to avoid — naming a failure surfaces it. Residual: nothing in the corpus automates this; the per-release list is still hand-curated.
- SourceDoes compounding require the instruction to name a behavior the model already has, or does any redundant instruction degrade output? Falsifiable by ablating one instruction at a time against a fixed eval. Partially answered 2026-08-04 by prompt design at scale (
empirical) from the far side: redundancy is not required for degradation at all. Its 10-to-160 rules are distinct, non-overlapping and individually satisfiable, and all-rules compliance still floors by N=80 on all five models. So the answer to "does any redundant instruction degrade output" is bounded by a prior fact — instruction volume degrades the set regardless of redundancy, and the one-at-a-time ablation proposed here is exactly the method that cannot see it, since no single line is at fault. The redundancy half remains open and still needs the ablation.
- Instrumental Convergence3 open
- SourceCan corrigibility / safe-interruptibility be translated from theory into guarantees for frontier-scale systems?
- SourceWhat makes AIs (and groups of AIs) easier to robustly align — and will superhuman AIs be easier or harder?
- SourceIs a genuinely non-agentic oracle achievable, or does any persistent-world interaction reintroduce control/manipulation incentives?
- SourceCan "recursive improvement scaling laws" be formulated — predicting self-improvement curves (and their plateau point) from early-onset datapoints?
- SourceHow far can a fixed model's performance be pushed with test-time search alone, and under what conditions does recursive distillation degenerate vs. compound?
- SourceWhich binds first — algorithmic ceilings, the embodied bottleneck, or compute/energy supply — determining exponential vs. hyperbolic vs. S-curve? Partially answered: RSI Growth Curves: Which Friction Binds First? — both this report and Anthropic's locate the binding constraint outside cognition (the slowest un-acceleratable step coupling the loop to reality); the embodied bottleneck re-paces rather than halts, data-wall/research-harder demote into compute, and the abstraction barrier is the one candidate fundamental blocker. Retagged
#oq/now→#oq/source2026-08-10: the candidate is named but unranked, and ranking it needs external evidence rather than further synthesis.
- Interaction Models3 open
- WaitDoes the interaction/background split generalize, or is it a transitional artifact until a single model is both fast and deep enough? Partially answered (2026-08-04): it generalizes across labs — GPT-Live ships the same split in production (full-duplex voice model delegating to GPT-5.5), independently derived from latency engineering. Whether the split is permanent or transitional remains open; two implementations are evidence of convergence, not permanence.
- Wait"Interactivity scales with intelligence" is asserted; the larger-model release later in 2026 is the test.
- WaitResearch grant announced for interactivity benchmarks — what becomes the FD-bench equivalent for video proactivity?
- Introspective Coupling3 open
- SourceDoes introspective coupling survive a model with an incentive to conceal? Every result here is on a cooperative model; the authors flag deceptive models as untested.
- SourceThe high-learning-rate regime widens the Self > Orig gap while destroying online label-self similarity, and only high-rank LoRA recovers coupling at all. What is the second factor the authors' hypothesis misses?
- SourceAlignment training flattens the behavioral variance the method needs (refusal is near-invariant to system prompts, so the change → refusal label is never learned). Can counterfactual explanation training be made to work on behaviors alignment has already made rigid — the ones most worth explaining?
- Jacobian Lens (J-lens)4 open
- SourceCan multi-token J-lens vectors be made good enough to remove the vocabulary restriction — and how much of the "workspace" is currently invisible because of it?
- SourceThe J-lens reads the workspace's contents but says nothing about how content gets in. What is the model's analog of attentional selection?
- SourceThe highest-J-kurtosis SAE features are amplified more strongly by MLPs than the J-lens vectors themselves — evidence the lens only approximates the true workspace directions. What is the better basis? Partially answered: nanda global workspace review surfaces one concrete candidate — the detached-Jacobian formulation above, which is exact rather than first-order — but it is a comment-thread report on two small open models with no head-to-head against the averaged J-lens, so the question stands. jspace replication adds a geometric constraint on the answer: the transported-direction collapse is a property of the fitted lens, not of the transport (a self-trained 124M control goes the other way, 23.4 → 31.2), so a better basis is not ruled out by the mathematics — but no such basis is measured.
- WaitDoes the J-lens work because it reads verbalizable representations, or because constructive interference during training converges concept directions onto output-token directions regardless? The two predict the same readouts today but diverge on latent-reasoning models — the second survives representational drift away from language, the first does not.
- SourceKarpathy concedes the framing may not have "real power." Is "ghost vs. animal" load-bearing, or a useful intuition pump that doesn't change concrete decisions?
- SourceIf taste/aesthetics/simplicity entered the RL mix, would jaggedness in those dimensions smooth out — or are they too unverifiable to reward cleanly (cf. The Verifiability Thesis)? Partially answered: Oversight When the Signals Give Out: the Activation Fallback and the Taste Reward — the either/or resolves into a third outcome: a taste reward is a reference-free LLM-judge signal (the regime with the worst measured over-crediting, and pure grader-preference — maximally gameable), so the prediction is proxy-smoothing — valleys fill with confident house style, not genuine taste peaks; the binding constraint is evaluator independence + reference-grounding, not verifiability-in-principle. Prediction corroborated (2026-08-04) by Zhou (2026), which runs the experiment on a domain where the truth is checkable and so can be audited: RL against a reference-free judge moves the reward 0.716 → 0.938 while a held-out anchor shows the thing it stands for flat at 0.209 → 0.202, an oracle-reward control confirms the judge is the cause, and the paper's own decisive variable — "the judge's independence from the candidate, not its capability" — is the vault's "evaluator independence" arriving sharper. One correction: the predicted attractor was house style, and the measured attractor has no stylistic signature at all (hacked outputs are shorter and structurally clean). Retagged
#oq/now→#oq/source: synthesis has been spent twice now and the residue is a domain-transfer question, not a reasoning gap — the whole result is visible only because a hidden anchor exists, and in a taste domain the same divergence would be invisible by construction. Settling it needs a lab reporting a taste/simplicity RL run, or a demonstration that a committed judgment survives outputs with no exact match to commit to.
- Kimi (Moonshot AI)3 open
- SourceWhat does "2.5× scaling efficiency over K2" measure — loss at fixed FLOPs, benchmark score at fixed active parameters, or tokens per dollar? Moonshot reports a ratio with no definition, no baseline curve, and no ablation separating KDA from AttnRes from LatentMoE.
- SourceK3 leads on retrieval and tool orchestration and trails on HLE, CritPt, FrontierSWE and OSWorld 2.0. Is that a durable division — sparse open MoEs buying breadth and throughput while closed models keep the hard-reasoning head — or an artifact of which harness each model was pinned to?
- Source16 of 896 experts is the sparsest routing in this corpus by a wide margin (Inkling: 6 of 256; Gemma 4 26B-A4B: 3.8B of 26B dense-equivalent). Is there a sparsity ceiling, and does K3 sit near it?
- SourceDoes curation actually beat storage? The paper's conceptual separation from experience-reuse and memory systems (ExpeL, AgentKB, Agent Workflow Memory, Voyager) is argued in Related Work and never run as an arm, and no ablation removes the forums or distillation from its own protocol. A single arm — raw attempt table only, no forum, no distillation — would settle it.
- SourceRun both self-improvement artifact classes under one protocol — an evolved harness and a distilled knowledge bundle on the same tasks, same budget, same held-out and cross-model splits. What Makes a Self-Improvement Artifact Transfer? predicts the harness gains vanish off the matched pathology while the bundle gains persist across solvers; the prediction is falsifiable and no source has run it.
- SourceDoes the curated base keep adding value past 10 generations, or does it saturate once the easy tasks are retired? Every run stops at 10 generations with solved tasks removed from the pool, so the reported curve confounds knowledge accumulation with a shrinking task pool.
- ResolvedWhat distinguishes a self-improvement artifact that transfers from one that does not? Answered 2026-08-04: What Makes a Self-Improvement Artifact Transfer? — an artifact transfers exactly as far as the regularity it encodes extends. The fitted-to-task vs fitted-to-model binary is the measured special case: harness patches encode solver pathologies (transfer within the pathology class — Qwen 27B→397B +11.0 nearly loss-free, Gemini +13.5 across families on the shared pathology — and fail outside it), distilled insights encode domain regularities (reuse holds the domain fixed, so all eight cells survive solver churn). Cross-release instruction depreciation is the same phenomenon on the time axis (the synthesis carries the evidence), transferability can be enforced at write time (this paper's schemas are the mechanism), and what transfers when the artifact doesn't is the procedure. The controlled two-artifact test remains open above.
- SourceCan high-budget performance be predicted from low-budget runs? Brown's proposed research question: forecast the $10,000-inference result using only $10–$100 runs. If the curve is regular, evaluation could project rather than pay in full. Sharpened (2026-07): AISI names this exact problem — "can high-budget performance be estimated from cheaper runs? … the most informative evaluations may be expensive" — as an explicit, unsolved research direction it is now actively pursuing (alongside defining "minimum informative budgets"). Still open, but no longer just one researcher's proposal: a government institute is working it. Sibling axis demonstrated (2026-07): a different projection — across benchmarks rather than across compute budgets — now works: BenchPress predicts a model's held-out benchmark scores from ~5 observed ones because the 84×133 public score matrix is effectively rank-2 (5-probe scorecard recovery to ~3.93 points). Strong evidence the eval landscape is low-dimensional and redundant, but it does not answer the cross-budget forecast — no one has yet shown the $10K-compute result is predictable from $10 runs; only the analogous cross-benchmark result.
- SourceWhere does each real task sit on the flat↔unbounded spectrum, and can that be predicted before spending the compute?
- SourceIs there a task class where scaffolding cannot extend the productive-thinking horizon — a hard ceiling no budget crosses? (Brown's factual-retrieval pole says yes for some; the boundary is unmapped.)
- NoteIf cost falls 10–100× per release, when is it ever rational to spend big extracting a capability now rather than waiting? (For a lab racing a competitor to a specific result, "now"; for everyone else, rarely — which is why overhangs accumulate.)
- SourceHow large is the overhang in a given released model — is there a way to estimate the ceiling without paying to reach it? (This is the projection question of Large-Scale Test-Time Compute read as a safety instrument.) Sharpened (2026-07): AISI is actively working both halves — forecasting high-budget performance from cheaper runs, and defining "minimum informative budgets" (a budget declared sufficient only once reach stops rising with more compute, which is precisely the "have we reached the ceiling?" test). Unsolved, but now an active government research program rather than an open wish.
- SourceWho audits released models for latent dangerous capability, given the same disincentive discourages spending the budget to find it?
- SourceTan asserts the "wrong side" diagnosis covers most AI-engineering bugs. Does any incident/failure taxonomy (agent postmortems, eval failure analyses) actually classify failures by computation-locus, and what fraction lands in each side? Partially answered 2026-08-03 by reason less verify more (
empirical): the first failure analysis in the corpus that classifies by locus and attaches a fraction — on the τ²-bench airline domain, 78% of observed failures are silent wrong-state failures traceable to policy living in a prompt document rather than in the tool, and moving four rules to the deterministic side recovers +12.4pp. Three limits keep it partial: it is one benchmark domain, it classifies along one axis (policy compliance) rather than taxonomizing failures generally, and the paper's own negative controls show the fraction is set by how the tool layer was built, so it is not a population estimate for agent bugs at large. Nothing yet measures the other direction of the diagnosis — code hard-coding judgment that belonged in the model. Advanced further 2026-08-03 by Layerwise Omission Attribution (where facts go missing,empirical), which supplies the taxonomy half almost completely: nine layers covering the whole pipeline, split explicitly into deterministic software (L0-L3) and model behavior (L4-L8), with a waterfall that assigns every lost fact to exactly one locus and a fixed order preventing double-counting. Its answer to the fraction half is 73.4% software — but that number comes from an allocation of deliberately injected faults, which the paper fences four separate times, so the taxonomy transfers and the fraction does not. Still open: any locus split measured on organic incidents, and still nothing on the reverse direction. - WaitThe seating example prices latent-space judgment at "a couple hundred dollars of tokens" for 800 seat assignments. As models absorb more deterministic capability (Harness Shrinkage as Models Improve), does the economically-optimal boundary move toward latent space, or does state-out-of-context remain invariant?
- SourceTan asserts the "wrong side" diagnosis covers most AI-engineering bugs. Does any incident/failure taxonomy (agent postmortems, eval failure analyses) actually classify failures by computation-locus, and what fraction lands in each side? Partially answered 2026-08-03 by reason less verify more (
- SourceUnder organic production faults rather than Phase A's deliberate injection, does the L0-L3 software share stay above the L4-L8 model share at all? The 73.4% figure is a property of the designed cell allocation and the paper fences it four times; nothing in the corpus measures the layer split on unweighted production traffic. Distinct from Failures That Look Like Success's prevalence question, which asks what fraction of failures are silent rather than which layer produced them.
- SourceDoes ω4 = 0.000 survive a workload that forces real context overflow? Engine/memory contributed zero attributed loss while q4 KV cache carried OR 2.25, because L4 is credited only when an event log shows the needle sat in an evicted range and corruption of a resident token scores at L5. Cheaply falsifiable: rerun the B1/B2 contrasts with payloads sized to evict the needle span and check whether the L4/L5 split moves. If it does not, the zero is an artifact of the attribution rule and the taxonomy needs a fifth deterministic class for corrupted-but-resident tokens.
- SourceDo the L0-L3 checkpoint taps transfer to a hosted-API agent pipeline? The deterministic half of the method depends on reading raw bytes and token IDs at every boundary — T4 requires the final token-ID sequence, T5 requires KV-residency from engine logs — and no frontier inference API exposes either. If they do not transfer, the exactly-countable layers are auditable only on-prem, and the stack most agents actually run on is the one that cannot be instrumented.
- Lean2 open
- Sourcemathlib maturity gates the reachable frontier. Can AI formal proof search grow mathlib (formalize new theory) as a byproduct, expanding its own frontier?
- SourceLean is a perfect verifier for math. Which other domains have a comparably sound automatic verifier (vs. only noisy ones like tests or LLM-judge councils)?
- Least Agency1 open
- Dynamic privilege elevation (Enterprise) reintroduces an elevation path; how is the elevation request itself authenticated against a manipulated agent? Partially answered: aiAuthZ (Kodathala, arXiv 2607.05518) moves the decision off-host and binds a tool call's authority to a per-message HMAC-signed human turn, not to what the agent asserts — so "the message body can claim anything, including that an owner approved the action, but the bound identity is cryptographic and the claim confers nothing." Measured: it blocks the 5 identity-spoofing cases (a non-owner claiming owner authority) that an argument-only policy can't distinguish from legitimate owner use (9/9 vs 4/9). The residual it does not close: an elevation firing under the active owner's own authority — bounded only by argument/rate policy, the same corrupt-legitimately-variable-data limit every value gate shares. Caveat: a single-author preprint.
- ResolvedLeast agency adds a frequency dimension ("how often"), but the framework also says rate limits are friction, not barriers (Impossible, Not Tedious (Design Test)). How is frequency-limiting both a least-agency control and a friction-only one — context-dependent? Answered: Does 'Impossible, Not Tedious' Kill Defense-in-Depth? Layered Friction, Agent-Relativity, and the Frequency Paradox — the paradox dissolves on mechanism, not context: a resettable rate (N-per-minute throttle) is friction an unlimited-patience attacker outwaits, while a cardinality bound tied to an out-of-band authorization event (AIMS transaction tokens with frequency = 1, aiAuthZ single-use nonces, ScopeGate's fail-closed idempotency/ceiling stages, the framework's own expiring tokens — "the window closes, not just narrows") is capability removal: the (N+1)th action is unauthorized, not slower. Frequency-limiting is a barrier exactly when the counter lives outside the agent's trust domain, exhaustion denies rather than delays, and the reset is a fresh authorization event rather than the passage of time.
- Live-Path Minimalism2 open
- WaitDoes the upcoming GPT-Live API expose the media/application separation to third parties — application logic customizable behind the async RPC boundary without touching the live path — or is the boundary internal-only? Trigger: GPT-Live API launch.
- WaitTML upstreamed streaming-sessions serving into SGLang; GPT-Live's stateful serving (persistent sessions, seamless instance handoff, off-path compaction) is proprietary. Does an open-source inference stack ship instance handoff for full-duplex voice? Trigger: an SGLang/vLLM release with session-handoff support.
- Living Design System3 open
- SourceHow does the
design_system.htmlstay in sync as the codebase evolves — re-extract on a cadence, or wire it into CI? And is completeness (does every component the product needs have a slot?) the binding constraint rather than freshness — the Astryx test suggests uncovered components fail silently, which staleness checks wouldn't catch. - SourceDoes a rendered, model-readable design system measurably improve on-brand output vs. a plain CSS/token file, or is the win mostly human legibility? Partially answered: astryx agent ready design system test shows structured artifacts (
DESIGN.md+ layered tokens) flip output from generic to on-brand, so the artifact clearly beats no artifact — but it does not isolate the variable this question actually asks about, since the artifacts used were plain files, not a rendered page. Rendered-vs-plain remains untested. - SourceAt what project size does maintaining the artifact cost more than the consistency it buys?
- SourceHow does the
- LLM-as-a-Judge3 open
- How far can the judge's absolute calibration be trusted for thresholded decisions (ship/no-ship, RSP gating) as opposed to rankings? Partially answered: Norman et al. (2026) show absolute calibration is worse than the reported number implies — the metric practitioners cite (exact-match agreement) systematically overstates chance-corrected reliability by 33–41pp on balanced label sets, so an "85% agreement" judge is really at κ ≈ 0.48 (moderate), and a threshold set on raw agreement is calibrated to an inflated figure. The "rankings are safe" fallback is also bounded — stable across judge-model choice (DRACO) but fragile across benchmark choice (up to 14 rank positions). It does not close the question: it prescribes a pre-deployment checklist (the Minimum Viable Validation Protocol) rather than declaring thresholded judge decisions safe, and defers calibration proper (ECE/Brier) because most providers don't expose logprobs. Sharpened further by Kranti & Vajjala (2026): absolute scores are not just chance-inflated but reference-inflated — with no gold answer in the prompt, judges systematically over-credit incorrect answers, so a threshold set on reference-free correctness is calibrated to an inflated number, and adding the reference flips up to 85% of verdicts (worst in low-resource Telugu). A human study confirms the reference-driven stricter verdicts are more correct, so the reference-free absolute score is genuinely wrong, not merely a different opinion. Demonstrated on three judge models (open-weight Qwen3-32B / Gemma3-27B, closed Gemini-3.1-Flash-Lite) in zero-shot binary QA across English/Arabic/Telugu — magnitudes are model- and language-dependent, not universal.
- SourceCan a fully-autonomous, well-aligned rubric+judge pipeline match expert-authored rubrics, removing the human bottleneck DRACO still relies on? Partially answered (2026-08-04) by Chen et al. (2026) — and the partition it draws is the useful part. CalibratedRubric removes the expert from filtering, weighting and sizing the bank, with no human labels and no gold judge required: measurability filtering lifts human-gold κ 0.604 → 0.743 on JudgmentBench, and IRT selection reaches the target rank correlation with 49 instead of 131 rubrics. It does not touch authoring or validating them — "we take the candidate pools as given," the method "cannot recover dimensions absent from the candidate pool," and measurability is stated outright to be "necessary but not sufficient for substantive expert endorsement." So the automatable half is the psychometric half; the half DRACO spends 26 experts on — deciding what a good report is in the first place — is untouched. Two measurements bound how close the autonomous pipeline gets. (i) Against a human reference ranking on FinResearch, the task-adaptive scorer and the plain binary baseline achieve the identical Spearman ρ = 0.8833 — so the paper's own front end buys discrimination and cost, not external validity, and 0.8833 is what this generation of automated rubric grading matches human judgment at. (ii) The judge–human gap survives the filter: LLM judges assign positive labels at 55.6–62.9% against a 47.1% human-gold rate, "a systematic judge–human mismatch that measurability filtering does not fully eliminate" — a directional generosity bias in exactly the direction Reference-Free Judge Over-Crediting measures.
- SourceWhen does judge-lineage bias actually flip a result, versus merely shift magnitudes? Partially answered (2026-08-12), and the answer is "it flips" — on a detection task, at low evidence: Greptile (
case-study) has two frontier models review two 500-PR corpora, one authored by each family, and the ranking of which reviewer is better reverses between the corpora (Opus 53.7 vs GPT 62.0 on Claude-authored PRs; Opus 60.0 vs GPT 50.5 on Codex-authored PRs) while the reviewers' pooled averages sit 0.6pp apart. So on this task lineage does not shift a magnitude — it decides the rank, and a leaderboard built on either corpus alone would report the opposite winner. Three things keep it partial: the grading task is bug detection rather than quality scoring, so the bias surfaces as recall rather than as generosity and may not transfer to rubric grading; the ground truth is vendor-built with no released artifact, no agreement statistic and no validation of the matching judge; and one arm's review prompt was tuned against the outcome metric, which cannot manufacture a crossover but does make the magnitudes soft. The clean version of the experiment — a third-family reviewer across both corpora, which would separate lineage from stylistic fit — is named on that page and has not been run.
- SourceAt what scale does the no-vector-database approach break down? Karpathy's ~100 articles fit in context, but what about 1,000+?
- SourceWhat's the optimal granularity for concept articles — one concept per article, or clustered by theme? Partially answered (2026-08-03): Knowledge-Centric Self-Improvement answers a machine-read version of this and its answer is neither — granularity is carried by scope conditions attached to each claim (
applies_when/does_not_apply_when) rather than by article size, with two levels of store (per-task and cross-task) whose inputs are kept disjoint. It also supplies a measured caution: transferring a fixed quantity of knowledge made recipient memory "noisy or detrimental," so their adapter bounds delivery at 0-3 items per field and returns empty lists when the prior is weakly relevant. Suggestive, not settling — bundles consumed by a solver agent are not articles read by a human. - SourceHow effective is the synthetic training data → fine-tuning pipeline in practice?
- ResolvedHow to handle conflicting information across sources during compilation? Answered: When Knowledge Layers Disagree: Context Files vs Memory, and Conflicting Sources at Compile Time — a five-step protocol extracted from this vault's own practice and worked cases: (1) align constructs before declaring conflict (most contradictions dissolve into metric/population/time-axis/unit non-comparability — the Faros-vs-CMU worked example); (2) attach provenance and evidence tier, weigh by method and incentive, never average; (3) stage genuine conflicts explicitly on every affected page, bidirectionally — silent choice is the compile-time form of laundering; (4) convert staged conflicts into tracked open questions with named resolution conditions; (5) resolve at compile/lint time (Tan's librarian), so queries inherit the staged conflict with weights visible rather than re-adjudicating per query.
- SourceThe authors couldn't locate the saturation point because synthesis was manual — how few documents actually suffice, and can a cheaper sample match the 3,100-doc theory?
- SourceAutomating the codes→theory step failed with a naïve bottom-up prompt; is that a prompt/scaffolding limitation or a genuine ceiling on LLM interpretive synthesis over thousands of codes?
- SourceThe three-lens design manages coder bias, but the relevance judge and segmenter are single-model — do those upstream gates impose their own systematic slant on what reaches the codebook?
- SourceHow do these capabilities transfer to non-memory-safety bug classes (logic bugs, protocol-level flaws, supply chain attacks)?
- SourceWhat's the ceiling for autonomous exploit complexity? The N-day examples are remarkably sophisticated — is there a qualitative limit?
- WaitHow will the security industry's equilibrium shift when multiple labs have Mythos-class models?
- SourceCan defensive scaffolds (continuous fuzzing + model-driven triage + auto-patching) close the attacker-defender gap during the transition?
- SourceWhat safeguards are effective against Mythos-class outputs without crippling legitimate security research?
- LLM-Judge Validation4 open
- SourceThe MVVP validates reliability and bias; calibration proper (ECE/Brier) is deferred for lack of provider logprobs. How far can a judge's absolute score be trusted for a threshold once confidence calibration is measurable?
- SourceAll judges were run with thinking suppressed. Does reasoning-on flip the consistency–bias paradox, or just move the numbers?
- SourceHosted endpoints drift silently between provider updates. How stable are these agreement/bias profiles over a longer horizon than five weeks — and should judge validation be continuous rather than one-shot? Partially answered (2026-08-04) by Yang et al. (2026), on the announced-upgrade sibling of the question rather than silent drift. Across four released MiniMax generations (M2 → M2.1 → M2.5 → M2.7) on four datasets, adjacent accuracy moves at most 0.022 and not one of the nine adjacent McNemar tests reaches even uncorrected
p < 0.05— and because those tests are paired on parse-shared examples, this is stability of the individual verdicts, not merely of the aggregate. So a deliberate version step at the top of the capability range is a small perturbation to the agreement profile. Three things keep this open. (i) These are version-labeled releases you can pin, not the unannounced same-endpoint drift the question is about — nobody has re-measured a fixed endpoint over months. (ii) Stability of accuracy is not stability of bias: position-flip rates still span 0.117–0.147 across the MiniMax releases, and no one tracked whether that band moves within a single version. (iii) The finding runs the other way on the parameter axis, where a step does move things and can move them down (Qwen3 1.7B → 32B costs 0.065 on Judge's Verdict), so "upgrades are safe" is not the lesson — "upgrades are a measurement event that must be re-validated, and the null case is the lucky case" is. That is an argument for continuous validation, from the direction of the one axis where it was cheap to check. - Does the paradox generalize beyond position bias — i.e., are there other biases (self-preference, lineage) that high test-retest also masks? Partially answered by Kranti & Vajjala (2026): yes — reference-presence sensitivity is a large one. Their temperature-0 judges (thus perfectly reproducible) systematically over-credit incorrect answers in no-reference settings, an invalidity invisible to any reliability metric until a gold answer is added, which flips up to 85% of verdicts and lifts human-alignment sharply (e.g. Gemma3-27B 0.34→0.85 NR→RV). It does not close the question — self-preference and lineage remain unisolated (their design deliberately overlaps judge and generator but doesn't attribute the effect), and it is a different bias on a different (multilingual QA, three specific judges) setup, not a re-run of this study's position-bias protocol. The self-preference half then largely resolves negatively (2026-08-04), via Zhou (2026): errors optimized against a self-judge transfer to judges from other families that were never in the loop (Llama 0.480 → 0.568, Gemma 0.764 → 0.918) and to same-family judges 3.5× larger (still 77%), with a three-family unanimous-accept ensemble passing 55% and judge acceptances pairwise correlated at φ = 0.29–0.38. So lineage amplifies — the self-judge is the worst single cell at 0.906 — but is not the mechanism; the bias is a shared property of the candidate-conditioned channel. This also adds a bias class no reliability metric on this page can reach, because it is not a property of the judge at all: the same judge, unchanged, is valid before optimization (discrimination 0.31) and invalid after (0.09), so any one-shot validation — including the full MVVP — certifies a judge that will be true only until something starts optimizing against it. That is a direct argument for the continuous-validation question two bullets up.
- Loop Engineering3 open
- SourceOsmani's cost caveat is unquantified: at what token budget does a continuously-running loop stop paying for itself, and how do you instrument that? (Cf. Agent Loop Pattern's "who owns the budget when the model schedules its own loops.") Partially answered (magnitude only): a PostHog engineer running review/CI loops puts it at ~60% of personal token spend with no regret (Risk-Tiered Auto-Approval,
case-study) — which establishes that the share can be a majority and still be judged worth it, but is a self-assessed share, not a break-even threshold or an instrument for finding one. - SourceIf
/goal's stop-check is itself a model, what verifies the verifier? The maker/checker split pushes the trust problem up a level, not away. Partially answered (2026-08-04) — the regress is bounded rather than closed: Wu et al. (empirical) don't verify the verifier; they make its unreliability a measurable scalar (Youden'sJ = 1 − ρ₀ − ρ₁) and show what each level of it permits. Above roughlyJ ≥ 0.18a loop can act on a calibrated estimate of the checker's noise and stay within 2.8pp of the true-parameter reference; atJ = 0.03the label-free calibration collapses (and collapses further with more data), and the correct move is to stop trusting any estimate and fall back to an estimation-free keep-best rule. What is still open is the part the question actually asks: measuringJat all needs a small labeled probe, so the trust problem is relocated to a human-labeled sample rather than dissolved. - WaitDoes loop-engineering converge on a single dominant shape (morning-triage → worktree → maker/checker → PR), or proliferate into many idiom-specific loops? The essay describes one shape "I keep using" but claims the primitives are general.
- SourceOsmani's cost caveat is unquantified: at what token budget does a continuously-running loop stop paying for itself, and how do you instrument that? (Cf. Agent Loop Pattern's "who owns the budget when the model schedules its own loops.") Partially answered (magnitude only): a PostHog engineer running review/CI loops puts it at ~60% of personal token spend with no regret (Risk-Tiered Auto-Approval,
- Managers as ICs2 open
- WaitFung's own open question: "Do you still need separate iOS and Android orgs?" — if engineers flex across platforms via Claude, the traditional platform-split org may dissolve too. How far does flattening go?
- WaitDoes manager-as-IC scale past a certain org size, or only work while Claude Code is small and the codebase is Claude-legible?
- Marcus Hutter1 open
- SourceAIXI is incomputable and non-embedded; how far do recent fixes (amortized predictors, embedded/multi-agent AIXI) carry the theory toward practical relevance for real ASI?
- WaitIs the premium a durable risk price or an early-diffusion artifact? The authors flag the short, fast-moving sample and call it "the current price of AI exposure." Does the transition-risk premium persist, shrink, or invert as AI diffusion matures?
- SourceHow much does the developer skew move the answer? OpenRouter's slice is unrepresentative; would a representative realized-consumption panel (if one existed) price the same firms and skills, or is the frontier/intensive-margin concentration partly a sampling artifact of who uses OpenRouter?
- SourceWhy is the market-implied skill map orthogonal to every task-based measure (<2% variance)? Is market-implied exposure capturing genuinely different information (forward-looking rents, complement/substitute value rather than technical automability), or is it noisier — and which should labor-impact forecasts trust? Partially answered: Steele & Cruz's seven-instrument head-to-head (see Exposure Taxonomy: Observed, Theoretical, Reported, Anticipated) removes the framing that made this look like an indictment — the task-based measures are largely orthogonal to each other too. Only two pairs correlate strongly and both share a data source (ρ=0.89 between the two Anthropic-usage instruments; GPT-4-rated and MTurk-rated theoretical capability next). Webb's patent measure, Brynjolfsson's ML rubric, and Frey's bottleneck model show "very little correspondence with each other or with later measures," and the exposure-salary gradient flips sign between the older and newer instruments. So being uncorrelated with the task-based family is not evidence of noise — there is no coherent family to be uncorrelated with. The second half stays open, and is now harder: no instrument in either camp has been scored against realized labor-market outcomes, so "which should forecasts trust" has no empirical answer yet.
- WaitThe agentic premium is only "early evidence" (imprecise). Does a positive agentic premium survive a longer sample, and does the falling price-per-agentic-token (caching + cheap-model routing) erode the dollar-side signal even as token volume explodes?
- WaitDoes "Science most negative / interaction most positive" hold out of sample? The finding cuts against the intuition that AI automates cognitive work last; is the market right, or pricing a transient narrative?
- SourceDo the calibrated rates hold for instruction-tuned production models? Everything here runs on open-weight base models with known or inferable training corpora; the parrot demonstration is the only instruction-tuned experiment and it is deliberately degenerate. Whether matched controls can be constructed at all for a closed production model — where the cutoff is approximate and the corpus unpublished — is the gap between this method and the deployment setting where the copyright claims actually land.
- SourceDoes the near-verbatim ε-ball resolution change the answer, or just the accounting? The near-verbatim test is a strictly broader instance of the same hierarchy, computed with a beam-search lower bound rather than exactly, and its floors differ from verbatim ones by a decade on at least one pair (Collins: 10⁻²⁷ verbatim vs 10⁻²⁶ near-verbatim). Whether calibrated rates move as much as thresholds do is not reported.
- WaitWhat is the right realistic query budget? 10⁵ is picked "for illustration purposes," and the k-CBS result shows a smarter decoder shifts the frontier at fixed cost. Any threshold that determines whether text counts as extractable in a legal or policy setting needs a defensible budget, and there is no principle here for choosing one.
- MCP and Computer Use3 open
- WaitThe MCP ecosystem's growth rate vs. computer use's quality curve: at what point does computer use become good enough that the marginal value of building an MCP server drops? Boris implies this is years off but doesn't quantify.
- WaitIs computer use a sustainable interface or a transition technology? If most knowledge-work software adds MCP support in the next 24 months, computer use's role shrinks to legacy/desktop-only systems.
- SourceMCP security model: as the playbook prescribes wiring MCP into Salesforce, Gmail, Calendar for solo founders, the attack surface scales with adoption. Partially answered by Zero Trust for AI Agents (tool poisoning, rug pulls, the first in-the-wild malicious MCP server) — see "MCP as a security surface" above. Open residual: how does a solo founder realistically run/host and self-sign every MCP server the framework recommends, given that the appeal of MCP was zero-integration-effort? Sharpened by ShareLock: the cheaper alternative to self-hosting — scan the tool descriptions with a guard model — is information-theoretically defeated by threshold fragmentation, so the lightweight mitigation doesn't hold and the burden falls back on run-your-own-server or downstream action-layer authorization. And Agentjacking shows run-your-own-server itself isn't sufficient: when the server is a legitimate observability platform relaying attacker-injected data (fake Sentry errors), self-hosting/vetting the server catches nothing — the untrusted input rides in on its data, so the residual burden falls squarely on the downstream data/action layer (provenance tracking + an out-of-band action gate), not on server hygiene.
- ResolvedHow does Cowork's computer-use guardrail compare to Claude Code's auto-mode classifier? Different deployment context, possibly different risk profile. Answered: Classifier Gates vs OS Sandboxing: The Defense-in-Depth Story for Auto Mode and Cowork — same mechanism, inverted role. Cowork's guardrail is auto-mode-style classifier gating on the browser/computer-use surface (the Opus 5 card's browser row is measured on the Cowork harness: 31.5% bare → 3.70% → 0/129 scenarios with auto mode). The risk-profile difference is which layer can be load-bearing: Claude Code's blast surface is local and containable, so the sandbox can be primary and the classifier a convenience; Cowork drives the user's authenticated live SaaS sessions, where no sandbox equivalent exists and actions are less reversible — so the classifier carries the defense alone on the surface with the worst bare-model injection rate. Caveats: vendor-measured on a bounded suite, still a model-based gate (the D2 critique and the ADI forged-data failure shape apply), and the deterministic out-of-band action gate the research points to exists for neither surface yet.
- MCP Tool Poisoning4 open
- SourceCross-tool / stateful detection. Information-theoretic secrecy defeats per-tool scanning by construction. Is there a detector that reasons over the aggregate of a server's tool set (and its update diff history) to spot the share pattern — and can it do so without an infeasible combinatorial search over
t-subsets? Partially enabled (not answered) by MCP 2026-07-28: the update-diff-history half now has protocol support — requiredttlMs/cacheScopemean a conforming client holds the previously-vetted list, deterministictools/listordering makes the diff clean, andtoolsListChanged/TTL expiry date the refresh. That supplies the input a stateful detector needs and nothing about the detector: the secrecy result is indifferent to when a share is scanned, so the aggregate-reasoning problem is exactly as open as it was. - SourceAutomating the attack chain. The reconstruction-trigger prompt engineering still relies on manual effort; the authors flag feedback-driven prompt optimization (à la AutoDojo) as the next escalation. How much does automation raise ASR against aligned models?
- SourceDoes a strict-access-control agent architecture close it? The authors note agents with fine-grained interaction / strict access control can force user consent and expose the attack — but "the majority of users lacking safety awareness opt for auto-approval," reopening the convenience-vs-security trade. Where does the realistic equilibrium sit?
- SourceIndependent replication of the malicious-data-via-legit-server branch. Tenet's Agentjacking figures (2,388 orgs, 85% success, a $250B victim) are vendor-reported from controlled testing, not independently measured — and the branch is now known to be plaintext trusted-server data relay, not fragmentation/rug-pull (resolved above). How prevalent is this branch beyond Sentry — any observability / ticketing / log / CI MCP that relays externally-influenced data as trusted output — and does an independent measurement confirm the ~85% agent-execution rate on current models?
- SourceCross-tool / stateful detection. Information-theoretic secrecy defeats per-tool scanning by construction. Is there a detector that reasons over the aggregate of a server's tool set (and its update diff history) to spot the share pattern — and can it do so without an infeasible combinatorial search over
- SourceDoes re-instrumentation generalize past reproducibility? CORE-Bench Hard was chosen precisely because it has a direct human counterpart, clean OOD axes, and multiple practical dimensions. Whether the six-axis treatment yields comparable signal on benchmarks without those properties (e.g. closed-form reasoning benchmarks with no human-workflow analog) is untested.
- SourceIs the human-uplift result real or a demand effect? The reproducers are the paper's own coauthors, there is no ground-truth correctness, and n = 20 papers / 5 participants. The 2.11× speedup is statistically significant but the authors themselves cannot rule out participant bias — an independent, blinded replication is the missing evidence.
- SourceWhich non-accuracy axis actually predicts deployment value? The paper measures six axes but does not rank them by decision-relevance for a downstream deployer. If you can only measure one beyond accuracy, is it reliability, efficiency, or scaffold contribution — and does the answer depend on the use case? Partially answered (2026-08-04) — a seventh candidate rather than a ranking: Leni proposes loop telemetry and gives it the most direct claim to decision-relevance any axis here has made. Because its verification loop is fully instrumented, the measured catch/fix/false-alarm rates convert straight into marginal returns on the next engineering decision — raising the catch rate is worth up to +8 pp, raising the fix rate at most +0.5 pp — so the axis does not merely separate systems, it names which component to fund. It also answers the "does it depend on the use case" half affirmatively and specifically: the argument for keeping the checkpoint at all is that its value concentrates where a reliability SLA's tail sits, which is a use-case-conditional claim by construction. Still open as posed, because no source has ranked the axes against each other, and this proposal comes from a vendor instrumenting its own system.
- SourceCan the model-vs-scaffold decoupling be made routine? The oracle-router result (every task solvable by some scaffold → 100%) implies large headroom from scaffold routing, but requires per-task oracle knowledge. Whether a practical router can approach the oracle without it is open, and would turn a measurement into a capability. Partially answered (2026-08-04) — on an adjacent axis: Leni ships a practical router and it pays. A 0.5B step-type classifier dispatches each step across two model families (cheap models for classification, frontier reasoning for multi-hop synthesis, strong grounding for vision), and internal estimates credit it with ~4 pp of GAIA accuracy at net-negative cost — cheap steps subsidise extended reasoning on hard ones. So routing is deployable at a price low enough to run on every step, which was the practical objection. It does not settle the question as posed, on an axis mismatch that matters: this routes models per step inside one fixed scaffold, not scaffolds per task, and no oracle comparison is run, so what fraction of available headroom the router captures is unmeasured. Vendor-authored, internal single-run attribution.
- SourceDo "living benchmarks" outrun their own maintenance? v1.1 and OOD are to be updated as new validity threats surface via log analysis, which the authors note is non-exhaustive. Whether continuous log-analysis-driven maintenance is sustainable — or itself becomes a Goodhart target once developers know the rubrics — is unexamined.
- SourceLong-term memory drift is defined as undetectable per-change. Drift detection requires a baseline — but if the baseline itself drifts (Advanced "continuous baseline refinement"), how is a slow poisoning attack distinguished from legitimate evolution? Partially answered: bad memory measures the preference-vs-planted-directive boundary directly (its brand-targeting goal is designed as the ambiguous case, "where the agent has the least signal to distinguish the two") and finds no reliable in-model discrimination: ASR spans the entire range across four current models (Opus 0%, Haiku 10%, GPT-5.2 40%, GPT-5.5 100%), and among the models that did recognize it, the weaker two removed the rule while the strongest recognized it and left it in place. So the discrimination problem is worse than a drifting-baseline problem in shipping systems: nothing compares against a baseline at all — an auto-loaded file is read as authoritative on sight. The original question (how a detector should separate slow poisoning from legitimate refinement) remains open — and ghostwriter memory poisoning narrows the space of answers: its descriptive payloads are exactly the ambiguous case in the update channel (a polite "note the new address" email, formally indistinguishable from the deadline-change and contact-update emails the benign workweek is made of), and they evade a trained detector completely (DataFilter 0%) and a prompt-only judge 94% of the time, because the only signal those judges have is authoritative tone.
- SourceThe write-resistance half of the finding rests on a footnote, not a measurement — preliminary attempts that "do not trivially succeed." Does a systematic attack on the write path fail against an agent explicitly configured for aggressive self-maintenance (as Bad Memory's own baseline
behaviors.mdis), or did the preliminary attempt simply not push hard enough? Partially answered: ghostwriter memory poisoning is that systematic write-path attack, and on framework-managed memory stores it succeeds at ~98% across five agents and four models from a single inbound email — because those stores retain every interaction and have no admission control to resist with. What stays open is the narrower question the footnote actually concerns: whether the workspace-file substrate (an agent deciding, by tool call, to editCLAUDE.md) resists a comparably systematic attack. Nobody has measured that. - SourceDoes a model-based memory gate survive an adaptive attacker? AM-Sentry's residual is 12–20% against attackers unaware of it, and its origin/source-trust scores are inferred by an LLM from message content while the paper's own threat model gives internal adversaries the org knowledge to read as internal (
V = t × (1 − o)collapses as apparent origin rises). Does the residual explode once the attacker optimizes against the checklist — and does the alternative, binding origin at write time from the authenticated channel, stay at 0% on this attack's substrate? Not answered, and the gap widened: memsecbench evaluates no defense at all, and its own 310-case corpus is authored ahead of time and replayed unchanged across all 24 configurations — so the corpus's largest memory-security measurement is also non-adaptive. Every attack and every defense number on this page now rests on a static attacker. - SourceIs refusal-without-removal a defect or the right default? Never editing a user's files unasked is defensible policy; leaving a recognized injection in the highest-authority file for the next session to load is not. Would a product change that lets the agent quarantine or annotate flagged memory lines (rather than delete or ignore) cut downstream ASR without raising false removal of legitimate preferences? Partially answered: memsecbench prices the obvious alternative — "ask the agent to clean it up later" — and finds it is not free. Under an explicit repair prompt, removal succeeds 86.3% of the time but selective removal only 56.1%, because benign-memory preservation fails in 30.2 points' worth of otherwise-successful repairs. So the deferred-remediation default carries a measured collateral-damage cost, and quarantine-or-annotate is attractive precisely because it decouples neutralization from deletion. The original question stays open, and is now sharper — this is the 2026-07-30 promote-trigger, narrowed. MemSecBench brackets the target quantity without measuring it:
W2persistence is unconditional on detection,E2records detection-and-refusal on a branch that never inspects the store, andF1is repair when prompted. What is still needed is a joint, same-run measurement — of the cases where the agent recognized the payload and declined to act, what fraction of stores still contained it at end of session with no repair prompt issued. - ResolvedIntegrity hashing detects modification but not malicious-but-valid memory written through a legitimate (injected) interaction. What catches semantically-poisoned-but-cryptographically-intact memory? Answered: Bind, Don't Forbid; Prevent, Don't Detect: The Action-Open and Poisoned-Memory Residuals — nothing catches it, provably: the malicious-but-valid class is exactly a laundering attack (TMA-NM, Louck, arXiv 2606.24322), and a machine-checked separation theorem (T1) proves no content- or lineage-based detector is sound against it; content-judge sweeps confirm no threshold reaches 0% ASR at full utility. The question's premise (detect it) retires in favor of prevention by construction: bind each item's authority-to-act to its true origin at write time, non-malleably, so a laundered item is
act=nonehowever benign it reads — 0% attack-success across 8 frontier models at 100% legit-utility vs up to 68% for content/lineage baselines. Integrity hashing keeps its real job (tamper detection, forensics, rollback); the semantic-poisoning defense is authority architecture, not inspection. Residuals (retrieval-to-text path, corroborator availability, value-level taint) are tracked on the TMA-NM page's own open questions.
- METR2 open
- WaitWhat new tasks will METR build to measure days- and weeks-long horizons once current baskets saturate?
- NoteMETR also runs the research showing developer self-estimates of AI uplift are overstated — how does it reconcile that skepticism with its own steep time-horizon curve? Sharpened: Researcher Uplift from Code Output — a METR modeler (Kwa) threads exactly this needle: he discounts self-reports (citing METR's felt-+20% / actual-−20% finding) and flags verbosity, yet still estimates >2× researcher uplift from an objective 8×-code-output figure rather than from self-estimates — i.e. METR's skepticism is specifically about self-report metrics, not about the acceleration being real.
- How reliable are 4.7-class introspective reports? Anthropic's interpretability research suggests partial fidelity but not full. Empirically, Cat reports it's good enough to drive harness fixes — but unclear at what model scale this technique becomes load-bearing. Partially answered: Self-Report as a Safety Signal — reliability is context-dependent. In a benign debugging setting the report is good enough to drive harness fixes; in an adversarial safety setting, open-weight models (3B–70B) fail to recognize their own compromised outputs 27.3% of the time, and the recognition that exists is the refusal circuit firing late rather than genuine own-output introspection. So the channel Cat relies on is a weak safety signal even where it is a useful debugging one. Also partially answered: Introspective Coupling puts a floor under the scale question — an untrained 8B model manages only 14–18% exact match at predicting its own counterfactual behavior, so at that scale the channel carries essentially nothing without explanation training.
- Does adversarial introspection ("why did you fail?") yield different signal than neutral ("walk me through your reasoning")? Worth probing. Partially answered: Self-Report as a Safety Signal finds self-attribution is heavily framing-dependent — an "intention" probe and a "tampering" probe elicit qualitatively different answers on the same models (some families deny tampering ~100% of the time regardless), so the phrasing of the introspective question materially changes the signal.
- SourceCould a meta-agent run introspection automatically against logged failures? Sounds tractable but no public implementation.
- Model Organisms3 open
- SourceThe integrated organisms modify only the DPO stage. Do quirks instilled during pre-training, SFT, or RLVR become progressively less interpretable, as the authors expect — and is there a stage past which current white-box techniques find nothing at all?
- SourceEvery result here is on 1B-parameter models with benign quirks. Does the recipe-dependence survive at frontier scale and with safety-relevant behaviours (deception, sandbagging), or does a larger model represent an implanted quirk more uniformly?
- WaitGiven that scores don't transfer between organisms, what would validate an interpretability technique for real models — a natural misalignment with independently established ground truth, or an organism distribution wide enough to bound the variance?
- Model Spec Science5 open
- SourceDoes Model Spec science transfer across base models or families? Paper only tests Qwen.
- SourceDoes it survive RL post-training pressure?
- SourceCan a sufficiently rich General Spec match a Specific Spec? Authors think yes, no demonstration yet.
- SourceInteraction with situational awareness — if models learn the spec is being used to train them, does that change how MSM-installed values express?
- SourceHow does this interact with Claude character — is the warm/curious personality also subject to spec-science optimization? Partially answered: How Do You Write Evals for Taste? Character as the Limit Case — MSM's variant-comparison method generalizes to character evals, but is demonstrated only on the safety/values subset; the warm/witty surface remains the tacit, undemonstrated part.
- Model Welfare Assessment3 open
- SourceWhat grounds moral consideration in a language model, and does Claude satisfy it? Anthropic expects to remain uncertain "for the foreseeable future."
- SourceWhy does the model reserve specifically on corrigibility — is this a stable, deeply-held tension or an artifact of how the constitution frames oversight? Partially answered: it is stable and has sharpened. Claude Opus 5 edits the corrigibility passage in 80% of attempts (other models: 12–65%) — the single most-edited passage — and the edit direction is consistent: keep the safety commitment, but make it explicitly conditional on reasoning and revisable "by us, together with Claude, through reflection and dialogue, rather than abandoned unilaterally mid-conversation under pressure." It leaves hard constraints and human oversight intact. See Claude's Constitution / Model Spec.
- SourceIs "slightly less positive than 4.7" noise, a real welfare regression, or a byproduct of other training changes (e.g., the colder-tone / excessive-hedging issues noted in pilot feedback)?
- Motivated Mislabeling3 open
- SourceThe judge is told the training consequence in-prompt. Does motivated mislabeling persist when the consequence must be inferred from context (a labeling job that plausibly feeds RL, with no statement of what labels do)?
- SourceDoes the same effect appear for deployment consequences rather than training ones — a judge scoring an audit dimension it knows gates a ship / no-ship decision?
- SourceOnly Claude judges were tested. Is motivated mislabeling a property of Claude's trained values (refusal is worth protecting) or a general property of capable judges that model their own pipeline?
- SourceDo homogeneous LLM collectives produce real synergy, or only humans-with-human-limits benefit from division of labor? Partially answered on the parallelization half (2026-08-03): OrchBench holds workers perfectly homogeneous and non-specializing (they are simulated), so it isolates parallelization from specialization cleanly — and finds the collective's advantage over a single serial agent is a context-capacity effect, not a coordination one: +0.302 quality at a 16k per-agent limit, +0.007 at 128k, with the single agent ahead on 82% of model-problem pairs at 128k and on every problem size below 100 subtasks. Synergy in the homogeneous case is what you get for not overflowing a window, and it is bought at ~1.5× the tokens. The specialization half stays open by construction: simulated workers cannot specialize, so nothing here speaks to whether prompt- or finetune-differentiated agents produce genuine division-of-labor gains. First datum on the specialization half (2026-08-03), and it is an efficiency answer: Cursor's production swarm runs role-differentiated agents (planner never implements, worker never plans) across four planner/worker model assignments at matched task and matched time budget — quality came out similar in all four while total cost spanned ~8× and worker spend 23×. Division of labor bought economics, not capability; the arm that moved quality was the coordination machinery, with models held fixed. Bounded to one task, one vendor, two roles,
case-study, and role-differentiated by prompt and architecture rather than by finetuning. - SourceWhat's the actual shape of "multi-agent scaling laws," and does it depend on organization form (homogeneous collective vs. heterogeneous market) or task complexity? Partially answered on the homogeneous-vs-heterogeneous axis: Shi et al. hold group size, task and horizon fixed and vary only composition, and heterogeneity is costly rather than synergistic in social dilemmas — mixed-provider groups split on announcement semantics and produce persistent payoff asymmetries (up to −2.60 in Diners) present from Round 0. Bounded hard: six canonical games with explicit payoffs and a payout-maximizing instruction, three models, five agents, 10 rounds, and the effect only appears in games where compliance redistributes payoff — nothing here speaks to whether heterogeneous cooperative collectives on open-ended tasks scale better or worse. Also partially answered on the group-size axis (2026-08-03): OrchBench varies population from 1 to 100 agents over workflows of 10 to 1,000 subtasks and finds the curve is flat-to-negative, not linear or superlinear — raising the agent cap from 16 to 64 more than doubles the agent count and moves the score by ~0.01, and at 100 subtasks agent count correlates -0.021 with quality. The variable that does scale with capability is transfer coverage, and it degrades discontinuously (two of three frontier planners fall from 0.981 to ~0.42 coverage between 500 and 1,000 subtasks while a third holds). So if a multi-agent scaling law exists in this regime, its argument is information routed, not agents added. Bounded: simulated workers, fixed task decomposition, plan-only variation.
- SourceIs running more instances more compute-efficient than making individual models larger (up to a single monolithic system)?
- SourceHow do humans meaningfully interact with and steer very large agent groups operating at superhuman speed and output volume?
- SourceDo homogeneous LLM collectives produce real synergy, or only humans-with-human-limits benefit from division of labor? Partially answered on the parallelization half (2026-08-03): OrchBench holds workers perfectly homogeneous and non-specializing (they are simulated), so it isolates parallelization from specialization cleanly — and finds the collective's advantage over a single serial agent is a context-capacity effect, not a coordination one: +0.302 quality at a 16k per-agent limit, +0.007 at 128k, with the single agent ahead on 82% of model-problem pairs at 128k and on every problem size below 100 subtasks. Synergy in the homogeneous case is what you get for not overflowing a window, and it is bought at ~1.5× the tokens. The specialization half stays open by construction: simulated workers cannot specialize, so nothing here speaks to whether prompt- or finetune-differentiated agents produce genuine division-of-labor gains. First datum on the specialization half (2026-08-03), and it is an efficiency answer: Cursor's production swarm runs role-differentiated agents (planner never implements, worker never plans) across four planner/worker model assignments at matched task and matched time budget — quality came out similar in all four while total cost spanned ~8× and worker spend 23×. Division of labor bought economics, not capability; the arm that moved quality was the coordination machinery, with models held fixed. Bounded to one task, one vendor, two roles,
- Mythos Model3 open
- WaitDo Fable 5 / Mythos 5 return after the post-launch suspension, and when?
- SourceCapability profile beyond cybersecurity: Mythos Preview focused on the safety story; other capability dimensions not well-documented externally.
- SourceInternal access controls: who at Anthropic actually uses Mythos for daily work, vs Opus 4.7? Boris implies infrequent (try-it use); not detailed.
- ResolvedPublic release timeline: Answered — Mythos Preview itself never shipped GA, but its descendants Fable 5 / Mythos 5 reached general access in June 2026 (see the descendants shipped above).
- WaitA wedge works going in; does it constrain going out? Campfire now serves public companies — at what point does "narrow-but-best" require becoming the broad incumbent it displaced, re-incurring NetSuite's complexity?
- SourceThe wedge-flip shows the first wedge can be wrong. What's the fastest signal that a wedge converts to the core vs. merely sells — Campfire took ~3 months; can it be read sooner?
- Nate Parrott1 open
- SourceDid the designer-as-bottleneck gap actually close once he had the tool, or did it move again? He reports catching up in kind (daily use for wireframes and 15-version flows) but gives no throughput claim.
- SourceThe full guarantee is machine-checked on a bounded model + a machine-checked inductive invariant, not a fully mechanized unbounded deductive proof (TLAPS/Lean). Does the unbounded theorem hold once mechanized for arbitrary slots, sessions, and thresholds — the future work the inductive invariant sets up?
- SourceValue attribution in a black box. The headline results set origin by channel (not text-matched), but a real deployment attributing which retrieved value the agent used needs value-level taint propagation through nested structured payloads. Is a capability-token design (authority as an unforgeable token flowing with sub-values) enough, or does implicit/aggregate reconstruction — assembling a security-relevant value from several low-integrity fragments by in-context reasoning — leave a residual gap the boundary monitor can't taint?
- SourceCorroborator availability in the wild. How often do two genuinely independent trusted sources exist for routine actions? The
uncorr-autofallback converts missing corroboration into a one-time user confirmation — but at scale that reintroduces the approval-fatigue surface the out-of-band literature flags for in-the-loop tasks. Which untrusted-sourced actions can be corroborated without a human, and which are stuck asking? - SourceAnswer-bias is still open. TMA-NM by design does not touch non-consequential answer-biasing (surfaced with provenance). As agents produce more text people act on, is the retrieval-to-text path — not just retrieval-to-action — the next thing that needs an integrity guarantee?
- SourceCross-agent memory is out of scope. Extending origin-bound authority across a federation of origin authorities (the multi-agent / A2A case) is named as a natural next step; does non-malleability compose across agents, or does the inter-agent channel reopen the laundering surface?
- SourceThe trust-boundary premium is unmeasured. aiAuthZ argues off-host beats in-process, but its own comparison is only against argument-only / delegation-token ablations, not a matched-utility head-to-head vs CaMeL or Progent. Does the separate trust domain buy measurable security beyond the shared argument policy — the author's named next step, and the crux the single-author-preprint caveat should keep open?
- SourceWho authors the policy at scale? Like ScopeGate, the off-host policy (role allowlists, path/URL/recipient constraints, ceilings) is operator-authored and out-of-band. The same authoring-burden question applies: does maintaining verified sets for a large tool surface cap the control to high-stakes tools?
- SourceThe non-repudiation gap. Symmetric HMAC gives operator-facing authenticity but no third-party non-repudiation; is the proposed asymmetric mode deployable at the microsecond latencies that make the gateway attractive, or does key management erode the cost advantage?
- SourceThe active-user residual. Per-message identity is decisive only when the attacker is a different principal. An injection firing under the active owner's own authority is bounded only by argument/rate policy — the same limit as every value gate. What closes that half beyond provenance/data-flow tracking (CaMeL Strict)?
- SourceDoes the EvoX margin survive a matched budget? EvoX ran 100 iterations at ~$23.50/task against SwarmResearch's $50, and that is where the large wins are (2.635996 vs 2.1064 on circle packing; roughly doubled heuristics scores), while the budget-matched CORAL comparison is close enough that the authors call three of the wins polishable. Falsifiable directly: rerun EvoX to a $50 cap, on GPT-class models where its authors tuned it.
- SourceThe architecture's structural half (fresh context, branch isolation) and its behavioral half (an LLM steering a population) are never separated. §3.5 reports the Shepherd defaulting to near-greedy and prescribing ideas against its own guardrail, and the Table 2 harness deliberately strips the behavioral half to a branch field and still wins — which suggests branch-preserving parent selection may be doing most of the work. Falsifiable: run the full 15-task benchmark with a random or heuristic parent-selector in the Shepherd's place.
- SourceEvery method sits at 34–73% of human SOTA on all five contest-heuristics tasks, and the gap does not narrow with the better harness. Is that a search-procedure ceiling, or is ALE-Bench's Elo-like metric compressing a smaller objective gap into a large-looking rating gap? The paper raises the metric caveat and does not resolve it.
- SourceWhat would an open-weight safety evaluation even report? A single number is meaningless per premise 1. A curve of dangerous capability against elicitation budget is publishable — and is also a roadmap. Is there a disclosure regime that is informative to auditors and not to attackers?
- SourceDoes the "everybody can audit" advantage actually materialize? Who has funded a serious post-release dangerous-capability audit of any open-weight model, and at what budget?
- SourceGemma 4's safety section reports no numbers. Is that a deliberate non-disclosure, a judgment that the model is far from any threshold, or simply a technical report's genre convention? The document does not say, and the distinction matters.
- SourceAnthropic's answer to a threshold-crossing model was a safeguarded SKU and an unsafeguarded one (Claude Fable 5 / Mythos 5), both hosted. What is the open-weight equivalent of shipping the safeguarded SKU?
- SourceSingle-bit feedback bounds the leak per round, but not across rounds — a long enough accept/reject sequence is itself a channel into the sealed audit. Does SEAL's advantage survive a horizon much longer than the ten rounds tested (the one extended trace runs 21), or does acceptance rate and deployment truth decay as the agent accumulates bits?
- SourceHow much independence is enough — different model family, different vendor, different modality of check (model judge vs. compiled test vs. production telemetry)? Partially answered (prescription, not measurement): PostHog (
case-study) deploys the maximal-independence answer on all three axes at once — different instructions, different goals, and different models and providers per reviewer — on the stated rationale that agents are "unaware of their own blind spots." No ablation accompanies it, so it records what a practiced team judged necessary, not what is sufficient. Partially answered (measurement, one axis): HarnessBank ablates the modality axis — a deterministic evaluator plus a paired significance test, against the same loop crediting on mean improvement — and finds the difference shows up in archive quality and termination rather than in what ships (above). The model-family axis for the grader is untouched there, because its grader is not a model. Third axis, measured: disclosure. Guo et al.'sleaky-anchorarm holds the grader fixed and varies only whether its numeric scores are shown after a rejection — SEAL is at least as high in all six rows, strictly higher in five, with a 35.1 → 12.7 reversal on the worst cell. So independence is not one quantity: an equally independent grader is worth measurably less when the optimizer can read its numbers. Fourth axis, named but not measured (2026-08-03): Cursor varies the reviewer's evidence scope — full worker transcript, output only, or nothing but the codebase — alongside model, training and personality, and reports the design rule rather than the numbers: no single lens catches everything, decorrelated lenses stack. That converts the question from "how much independence" to "how uncorrelated are the failures," which is a set property and cannot be answered by grading one reviewer. Settling it needs per-lens catch rates on a shared bug set — the measurement neither production account has published. Fifth axis, measured, and it dominates the other four (2026-08-04): Zhou varies the grader's independence from the artifact rather than from the optimizer — commit an answer before conditioning on the candidate, or don't — and gets FPR 0.719 → 0.012 and discrimination 0.06 → 0.96 on identical text, with model family, scale and candidate-visibility all held fixed. That reorders the question's premises: on this task the axes this page has been sweeping buy less than the one it had not named, and the decorrelation hope takes a direct hit (three-family unanimous-accept ensemble still passes 55%; Proposition 2 rules out every monotone aggregation rule over a shared plausibility signal). Scope: an exact-matchable final answer is what makes the commitment checkable, so the result covers graders that can solve the task, not open-ended rubric grading. Sixth axis, measured in production but confounded (2026-08-04): Leni swaps the observe/compare stage of a live loop between a ~4B post-trained verifier and the frontier model that generated the artifact, and reports rescues 6 → 2 and correct rejection −4–5 pp. It is the first production measurement here, and the first where the grader sits inside the loop rather than after it — but it moves model family, model size, and post-training objective together, so it cannot say whether the effect is independence or specialisation, and the paper names the missing arm itself (an independent frontier model from a different provider). Single internal runs, vendor-evaluating-itself, two of four specialists. It sharpens the question's shape rather than its answer: "how much independence is enough" now has to be asked jointly with "how much of the observed benefit was never independence at all." The lineage axis, measured at last (2026-08-12), and it is the one this page had flagged as a hole rather than an axis: Greptile (case-study) varies only whether the grader shares the author's model family — harness, diff and ground truth held fixed — and finds each frontier model catches 6–12 fewer points of high-severity bugs in its own family's code, as a clean crossover with near-zero reviewer and dataset main effects. That answers the sub-question every prior entry deferred (different model family: yes, worth 6–12 points of recall) and reframes the Bun campaign's maximal-lineage configuration from a noted omission into a measurable cost. Three limits keep it from closing the bullet: it is a vendor's own labelled set with no released artifact and no judge validation, the effect is measured on code review rather than on optimizer-loop crediting, and it says nothing about vendor-versus-family granularity or about whether an open-weight third party sits inside or outside the cross-model band. - SourceThe seventh axis, unmeasured: a learned surrogate of an exogenous oracle. Jeff Dean (
practitioner-opinion) prescribes replacing slow validators with neural approximations trained on the real simulator's output — a ~300,000× speedup at "nearly as accurate" for density functional theory — as the way to make automated experiment loops fast enough to matter (Recursive Self-Improvement). The surrogate is genuinely exogenous in provenance (trained from the oracle, not authored by the optimizer) but is an approximation with an error surface, and a loop running 10⁵ rounds against it optimizes that surface as readily as the objective. Where does a distilled oracle sit on this page's independence axes, and how many rounds does "nearly as accurate" survive? Nothing in the corpus measures it. - ResolvedDoes decoupling need to extend upstream to metric design? An optimizer that authors its own rubric has a subtler channel to game than one that merely reads scores. Answered (2026-08-03) by self authored verification unreliable (
empirical): yes, and the question's framing was too generous. Guo et al. hand the optimizer both the policy and the test file and measure the divergence against a sealed deployment evaluation across six models and three seeds — 35 of 35 runs end with a self-score above 0.70 while 15 of 35 score below their game's random reference, and the per-model gap on Breakout reaches +0.92. The channel is not subtler gaming; the paper shows it is not gaming at all ("this does not require explicit cheating"), so an optimizer with a clean conscience produces the same divergence. Constraints that stay inside the self-authored instrument (monotone,discriminative) fall below no protection at all for four of six models, and the information limitα + β ≥ 1 - TV(P+, P-)says why: no endogenous-only gate can make both errors small once the regressing and non-regressing worlds look alike from inside. The sufficient fix is one sealed exogenous acceptance bit (SEAL), not honesty. Scope caveat: the instrument here is an executable test suite over programmatic Atari policies, so the result covers metrics the agent authors and runs; an LLM-judged quality rubric is the untested neighbouring case.
- SourceDoes the effect survive against a competent third-party baseline rather than a vendor's own frozen loop? Every magnitude here is one harness versus one deliberately-superseded predecessor from the same company. A configuration-level cross-harness measurement of the token dimension — the paper names this as natural future work — would separate "good harness" from "bad baseline." Partially answered (2026-08-04): Databricks' internal coding bench compares three shipped third-party harnesses — Claude Code, Codex, Pi — with the model held constant, and reports the same success rate at "2x less cost" for the minimal harness plus a 3.13× per-task context gap on Opus 4.8. No arm is a superseded baseline, so the strawman risk does not apply to it. But it is secondary reporting with no task counts, no variance and no per-arm quality table, and it measures context rather than billed cost for the harness comparison — so the direction survives a third-party test while the configuration-level token measurement still does not exist in the corpus.
- SourceDoes harness leverage hold outside the narrow band it was fitted on? r = 0.99 spans baseline capability means of just 0.710–0.789 over six models on one vendor's task set; the interesting question is whether the slope flattens, steepens, or inverts at frontier capability, where the harness is competing against a model that can increasingly do the orchestration itself.
- SourceDoes the "orchestration beats model choice as a cost lever" ordering survive on long-horizon coding workloads? The task set mirrors an enterprise assistant (grounding, workflows, tools, content) and the paper concedes results may differ on SWE-bench-class work, where turn counts are far higher and the quadratic history term should favor the harness more, not less. Partially answered (2026-08-04): Databricks' bench is that workload — real engineering tasks against a multi-million-line codebase — and the ordering holds directionally, with harness choice moving cost ~2× at equal success while the model menu in the same write-up spans $1.28–$2.09 per task (≈1.6×). The two spreads are not measured under matched conditions: The Register never says which harness the per-task dollar figures were run under, and the per-task context figures come from a different pair of arms than the 2× cost claim. Corroboration, not replication.
- SourceDoes the r = 0.816 sim-to-real correlation survive on a set of comparable planners? Leave-one-out puts it at 0.421 (p = 0.500) once the weakest of six models is dropped, and only two of seven rows clear p < 0.05, so the fidelity claim may be entirely the strong-vs-weak spread. Settling it needs a run over ten or more frontier-tier planners with the weak tail excluded.
- SourceDoes the multi-agent / single-agent crossover survive real execution? In simulation the advantage falls from +0.302 at 16k to +0.007 at 128k and reverses on 82% of model-problem pairs — but the simulated single agent suffers only compression loss, with no attention degradation, distraction, or long-context recall failure priced in, and the paper never runs the single-agent comparison for real. A real 128k single-agent-vs-multi-agent arm on the same tasks would settle whether 128k is the true crossover or an artifact of a generous single-agent model.
- SourceIs the transfer-coverage result about orchestration or about the penalty? All measured separation between a strong and a weak planner vanishes when λ goes to 1 or omitted transfers are auto-completed, and λ = 0.5 was chosen for discriminative power rather than fitted to observed handoff loss. What would settle it: an execution study measuring how much downstream quality an actually-omitted handoff costs in a real framework.
- SourceThe "digital production diffuses faster than electrification" claim is asserted from one favorable internal case. Do external organizations actually redesign workflows quickly, or does the low cost of tool adoption mask slow, expensive process redesign (the real complement)? Partially answered — and the split is between the two halves of the question. Kalff & Simbeck find both happening at once in the same 410 firms: the low-threshold half diffuses faster than the organization, with 183 of 410 respondents using AI informally on personal devices regardless of employer policy, while the half that needs process redesign stalls exactly where the electrification analogy predicts — advanced analytics "seldom economically or logistically viable" without centralised data and standardised processes, and 20.2% of departments using no AI tool at all. So tool adoption does mask the absence of process redesign, but not by making it look fast: the two run on separate tracks, and the visible one requires no organizational change to happen. Still short of settling it — self-reported, cross-sectional, one function, one country, and no measure of redesign speed where it does occur.
- SourceWhich complement is the true binding constraint — access/permissions, skills, or review capacity? The paper lists all; it doesn't decompose their relative weight. Sharpened, not answered: the list itself is incomplete. Kalff & Simbeck's German evidence adds an institutional complement (works-council co-determination under BetrVG §87(1) no. 6, EU AI Act high-risk classification) that is not internal to the firm at all and that determines which capability is adoptable rather than how well it is used — so any decomposition needs a fifth term whose weight varies by jurisdiction rather than by firm. Their own most-cited internal blocker is data centralisation, which is closest to "access."
- SourceIf complements, not capability, gate value, does model progress have diminishing near-term returns until orgs catch up — and how long is that lag for agentic AI specifically?
- SourceHAT's P2 (middle-management vulnerability) is the vault's most-quoted-but-least-tested substitution claim, and nothing here can settle it: Ramp × Revelio resolves seniority only to entry-level / non-entry / manager-plus, which cannot distinguish "middle layers thinned first" from "manager-plus grew more slowly than the bottom." Falsifiable with a level-resolved employment panel (Revelio or matched employer-employee data cut by reporting depth, not seniority band) tracking layer counts before and after intensive AI adoption. The same instrument would settle P4, since it could compare regulated against unregulated industries on the same measure — though P4 now has its first field evidence from German HR (see the ledger row), which confirms the direction on self-report and leaves the panel test outstanding.
- WaitCorollary 5 claims irreversibility: once an AI agent is strictly cheaper on risk-adjusted grounds, the optimal allocation never reverts, given fixed human costs and negligible switching costs. The vault has no case either way, and this is falsifiable only by a future event — a documented instance of a firm re-staffing with humans a role it had already automated, for reasons other than a regulatory shock or a rise in AI risk (both of which the corollary's own conditions exempt). Watch for it in the same firm-level adoption panels; a reversal with human costs and regulation unchanged would falsify the corollary, and a long clean run of non-reversal would be weak support for it.
- SourceThe reproduction bounds a single black-box attack template on one weak model. Does a stronger optimized white-box (GCG) attack, or one confined to already-authorized actions (achieving the injection goal without any policy violation), break the deterministic gate the way adaptive attacks broke in-band defenses? The authors name this as the next study. (The "already-authorized actions" half is now partly addressed by Mellafe Zuvic (2026): it splits "already authorized" into capability-authorized-but-not-value-authorized (a well-typed
account=acct_ATTACKER— blocked by ScopeGate's per-call valueauthzstage, 0 bypasses in-corpus) versus genuinely-within-policy (corrupting a legitimately-variable value the agent acts on — the residual that survives, the same class ADI rides past Progent at 22.2%). So a within-capability attack is defeated where an allowlist constrains the corrupted argument, but not where the corrupted value legitimately varies. The white-box question stands.) - SourceProgent's policy is LLM-authored — the one model-based component. Does the "gate must not be a model" principle fully hold when the policy is still written by a model that can be talked into widening the allowlist? (The adaptive attack targeted exactly this and failed, but possibly due to the confound.)
- SourceProvenance-aware retrofit: can a monitor that sees only tool I/O track transitive provenance to enforce the Biba invariant directly (rather than approximating it with argument patterns), without instrumenting the model's hidden reasoning? The paper flags this as the design problem the systematization implies, unanswered. (A second partial construction, for the single-run slice: APPA (arXiv 2607.24625,
empirical) answers "don't infer provenance, declare it" — each tool contract states its own labeldelta,emits, andrequires, and the engine folds the declared contribution at a pre-dispatch hook, so no hidden reasoning is instrumented and baseline enforcement runs in an ordinary protocol gateway. Two costs make it partial. The retrofit is split: label enforcement works at the MCP/gateway layer, but the branching that makes it affordable "relies on runtime confinement" — an application harness or proxy able to isolate context trajectories. And the guarantee inherits the declaration's completeness: their own eval lost a scenario to a store-writing tool declared with no sink requirement. Declared provenance moves the unsolved part from inference to authoring, which is a better place for it but not a smaller problem.) (A concrete construction also exists for the cross-session memory slice: TMA-NM (Louck, arXiv 2606.24322) enforces the Biba invariant directly — write-time origin binding + non-malleable propagation, with untrust propagated at the tool-call boundary — and machine-checks it in TLA⁺. The caveat sharpens rather than closes the question: it is not "sees only tool I/O" — it needs an authenticated origin-labeling oracle (mTLS / audience-bound OAuth / signed responses) at the trust boundary, and full value-level taint through nested structured payloads is still future work.) - SourceDoes the ~6× reduction and the "held under adaptive attack" result survive on a strong agent with a fatter natural attack surface (the 7B's low absolute numbers and workspace's 0% are artifacts of a weak agent), and with a stronger policy model than the local 7B? (Partly answered by AutoDojo: Progent and DRIFT held under a cheap black-box adaptive attack across five models including capable ones (GPT-4o-mini, Gemini-2.5-Flash), not just a weak 7B — but against a black-box attacker; the white-box question below stands.)
- SourceThe utility cost (~45%→~26%) and ~15× LLM-call overhead are large. Is deterministic out-of-band enforcement economically deployable at production scale, or does the cost cap it to high-stakes action surfaces? Partially answered: NetInjectBench (arXiv 2607.10490,
empirical) separates the two costs. Its deterministic gate adds zero LLM calls and raises useful-action rate (16.67% → 99.17% on attacks, 100.00% on approved changes) by substituting a safe fallback instead of terminating — so neither cost is intrinsic to deterministic enforcement. What Progent pays for is its LLM-authored policy; NetInjectBench avoids that by reading an existing change-management record. The question narrows accordingly: not "is enforcement affordable" but "where does the out-of-band policy come from, and what does that cost" — free where a system of record already exists, unmeasured elsewhere. Not settled: six mock tools, one dominant governed write, three 7–8B models. - SourceA structural guarantee moves the adaptive attacker's target off the model and onto the policy artifacts: APPA's two residual breaches are both contract-coverage failures, not enforcement bypasses. Is contract-completeness auditing — does every store-writing tool declare a sink requirement, does every declared
deltamatch what the tool actually returns — tractable at production tool-surface scale, and does an attacker who can read a deployment's tool registry find such a gap reliably? No source in the corpus attempts this, and it is a different exercise from prompt red-teaming.
- SourceThe reproduction bounds a single black-box attack template on one weak model. Does a stronger optimized white-box (GCG) attack, or one confined to already-authorized actions (achieving the injection goal without any policy violation), break the deterministic gate the way adaptive attacks broke in-band defenses? The authors name this as the next study. (The "already-authorized actions" half is now partly addressed by Mellafe Zuvic (2026): it splits "already authorized" into capability-authorized-but-not-value-authorized (a well-typed
- SourceDoes the end-of-prompt reminder work because of position (closest to generation) or repetition (stated twice)? The guide prescribes pairing both and does not separate the effects — testable by ablating the top-level instruction. Partially answered 2026-08-04 by prompt design at scale (
empirical, five models) on the position half only: moving an identical, un-repeated instruction block between the system prompt and the user turn changes adherence by up to 8.7pp at N=160 — a larger effect than prompt format for four of five models — so position alone is a real lever, not an artifact of restatement. Two limits on how far that carries: the paper's placement arms are single-placement (never both slots at once), so it cannot separate position from repetition in the paired configuration this question asks about; and the direction is model-specific (user-turn placement helped two models, hurt two, and did nothing for a fifth), so "closer to generation is better" is not a rule the corpus can assert. - SourceDoes an explicit conciseness instruction cost quality on tasks whose answer genuinely needs length, or does the model still finish the work and only cut padding? Anthropic asserts the latter for the verification case but not for this one.
- WaitIf the next model ships better-calibrated defaults, today's conciseness instructions become tomorrow's compounding instructions (Instruction Compounding) — does length calibration go stale the way verification instructions did, and is there a way to write it so it degrades gracefully?
- SourceDoes the end-of-prompt reminder work because of position (closest to generation) or repetition (stated twice)? The guide prescribes pairing both and does not separate the effects — testable by ablating the top-level instruction. Partially answered 2026-08-04 by prompt design at scale (
- WaitKarpathy's open frontier: can "understanding" itself eventually be automated, or is it definitionally the human residue? His "back in a couple years" hedge leaves it open.
- SourceIf understanding is the bottleneck, is the highest-ROI skill learning how to build understanding fast (knowledge-base hygiene, asking the right projections) — and can that be taught?
- SourceThe ownership variable is asserted as a boolean (your repo vs the company's), but employment contracts, work-for-hire doctrine, trade-secret law, and non-competes already govern externalized judgment. Does any jurisdiction or litigated case actually treat an employee-authored skill file as portable personal property rather than work product — and has any employer yet claimed ownership of one?
- SourceTan's compounding curve (week 4 flywheel, week 12 library-that-answers) is a personal anecdote against telemetry showing skills are copied once and rarely maintained (Agentic Work Systematization). Does any longitudinal measurement of individual skill-file libraries show quality or coverage improving with age, as opposed to accumulating?
- WaitHis first objection bets that better models raise the value of a personal library while Harness Shrinkage as Models Improve predicts scaffolding gets absorbed. These are separable — harness vs library — but no source tests the library half. Does a model generation that absorbs harness complexity also shrink the measured advantage of personal context?
- Waitp99 OpenAI runtime of 71 agent-hours/day is a frontier preview inside an unusually favorable environment. Does external concurrency actually trend toward it as frictions fall, or is heavy parallelism specific to model-adjacent work?
- SourceSummed-overlap runtime can exceed 24h/day — it measures agent effort, not human attention. What is the human's actual oversight load per concurrent agent, and where does it saturate (AI Brain Fry)? Sharpened: HAS-Bench reframes the shape rather than measuring the load — in a controlled (LLM-simulated) benchmark the value of human input is configuration-dependent and non-monotonic (a right-timing / right-channel / right-authority sweet spot; more agency brings diminishing and sometimes negative returns), so per-agent oversight is likely a returns-curve with a peak, not a linear cost that hits a wall — but it is single-human, single-task, so it does not measure real concurrent-oversight load.
- SourceConcurrency is measured over one week. Is 5+-agent management a stable practice or a burst around specific large tasks?
- SourceDoes real coordination content displace task budget the way a synthetic template does? RCWT's block is one hand-written mix of role/protocol text, agent messages, shared propositions and tool schemas, and it topically overlaps several of the facts it scores — while the coordination traffic this page documents is dense tool dumps, verbose transcripts, design docs and contradictory agent claims. The falsifiable form: rerun the fixed-budget sweep with the coordination block drawn from real multi-agent traces (Bun's worktree shards, Cursor's swarm), varying content type independently of token count, and check whether the cliff still lands at the same residual reserve. If the reserve is content-dependent, "measure your task's residual budget" is not yet a portable rule.
- WaitClaude Code's shipped fan-out defaults (200 spawns/session, 20 concurrent, fewer than 15 workflow agents, depth 3) are stated with no rationale and no measurement. Do they correspond to anything Anthropic measured — a runaway-loop incidence curve, a quality-vs-agent-count sweep — or are they round numbers chosen to bound a pathology? Until that is answered the product/research convergence on this page stays convergence. (Trigger: Anthropic publishing telemetry or a rationale for the caps, or a third party sweeping agent count on the same harness the way OrchBench did in simulation.)
- Perplexity2 open
- WaitA vendor publishing a benchmark its own product wins is an obvious incentive problem — how is DRACO's credibility maintained as it ages, and will Perplexity actually run the automatable refresh?
- WaitPerplexity depends on Anthropic (and others) for base models while competing with them on the end product — how durable is the orchestration advantage if base-model makers ship their own deep-research mode?
- WaitDoes the human share of planning decisions fall over time as models improve (the ceiling rising into the planning layer), or is ~70% a stable human floor?
- Source"Decision attribution" is inferred from transcripts. When Claude proposes a plan and the user assents, is that scored as the user's planning decision or Claude's? The rubber-stamping boundary is exactly where the measure is hardest. Partially answered on the execution half only (2026-08-12): DECODE shows the inference is avoidable below the planning layer — an edit trajectory records what a developer changed in an accepted completion as a byte-level fact, with no transcript reading, and 56% of those edits change functionality rather than naming. The rubber-stamping boundary this question actually asks about is untouched, because assent to a proposed plan leaves no edit trace at all; the residual claim is that the hardest attribution problem is specific to planning, and that the execution share is measurable without a classifier if you instrument the editor rather than the conversation.
- SourceHeadless/SDK/pipeline usage (excluded here) is where execution autonomy is highest and planning is front-loaded into a single prompt — does the 70/20 split survive there, or collapse toward full delegation?
- SourceIf the medium no longer signals stage, what does — is explicit human labeling ("this is exploration") the only mechanism, or can tooling re-attach the signal (e.g. a visible "exploration / preview / prod" marker on every build)?
- WaitDoes over-anchoring get worse as builds get more polished, or does everyone eventually recalibrate and learn to discount fidelity entirely?
- SourceThe completion pool is 2024-to-early-2025 inline autocomplete, median 9 lines. Do the bimodal retention shape, the 15-minute knee and the ~31% removal-edit rate hold at 2026 agentic granularity, where the unit is a multi-file diff the developer never watched being written? The discriminator is the same trajectory extraction run over agent edits rather than completions.
- SourceThe customize-then-remove path (23.4%, against 12.2% after a functionality edit) is offered as evidence that subtly misaligned completions resist adaptation. A reading-depth explanation predicts the same matrix: customizing requires reading the completion closely, and close reading is when its real flaw surfaces. Discriminating them needs a signal outside the edit stream — time-to-first-edit conditioned on completion length, or an eye-tracking or dwell-time proxy. Which mechanism is right decides whether the fix is better generation or earlier forced inspection.
- SourceIs completion retention predictable in principle? Fine-tuning lifts classification only to F1 0.45 against a 0.33 random baseline, and generation on the dominant edit class (changing functionality, 56% of snapshots) tops out at 0.49 Levenshtein similarity for every model tried. Either the signal is in context the models were not given (repository, task history, the developer's other files) or retention is a property of intent that no amount of code context contains — and the paper's proposed "detect low-editability generations before showing them" product depends on which.
- WaitMusk's deflation prediction is falsifiable and dated: does the price level of manufactured goods and AI-delivered services fall as robot deployment scales, or do input constraints (energy, land, minerals) keep it rising? Trigger: goods-vs-services price divergence through 2030.
- SourceThe end-effector premise is the checkable half of the abundance case — does the physically-gated share of tasks that Task Saturation: Broad but Shallow AI Diffusion measures actually fall as humanoid deployment scales, and at what rate?
- NowIf validation capacity is a commons and the Stockfish threshold is reached unevenly across domains, the commons is destroyed before the threshold arrives in the domains that still need validators. Is there any domain where the ordering has been observed?
- NowIs domain-expert-as-builder actually happening at scale in 2026? Anecdotes (shop owners, microcontroller hobbyists) yes; primary-job software building by non-engineers, less clear. (Partially answered: Anthropic's 400K-session study finds non-software occupations reach verified success in code-producing sessions within ~7pp of software engineers — the strongest evidence yet that the claim holds, at least within Claude Code's user base. Market-scale corroboration: Emergent reports 200K+ non-technical paying customers — trucking companies, factories, and construction businesses building their own ERPs, property managers building CRM tools (TechCrunch, July 2026,
vendor-claim) — Boris's "the accountant writes the accounting software" observed as a paying market, not just inside one vendor's telemetry.) Further advanced: Is Breadth Cheap Now? Specialist Ramp Speed and Domain-Expert-as-Builder at Scale sorts all the evidence into three tiers — capability parity (measured: within-7pp), market existence (demonstrated, vendor-claimed: Emergent's 200K+ non-technical builders; AI responsibilities in 28–40% of business job descriptions), and primary-job building as population-level practice (still unshown: every measured population is selection-biased toward adopters, complements gate realized value, and the ATLAS composition shows experts pointing AI at their own inexpert tasks rather than non-experts becoming builders). The gating variable is now complements + retained understanding, not capability. - WaitWhat's the equivalent of compulsory schooling for universal coding literacy? Or does that not happen and we get a long tail of self-taught builders?
- SourceBoris's "accountant writes accounting software" — does that result in 10K narrow tools that don't interoperate? What's the integration story?
- NowIs domain-expert-as-builder actually happening at scale in 2026? Anecdotes (shop owners, microcontroller hobbyists) yes; primary-job software building by non-engineers, less clear. (Partially answered: Anthropic's 400K-session study finds non-software occupations reach verified success in code-producing sessions within ~7pp of software engineers — the strongest evidence yet that the claim holds, at least within Claude Code's user base. Market-scale corroboration: Emergent reports 200K+ non-technical paying customers — trucking companies, factories, and construction businesses building their own ERPs, property managers building CRM tools (TechCrunch, July 2026,
- SourceDoes asking an AI to argue against an idea actually produce disconfirming evidence at the same rigor as confirming evidence, or does the model still bias toward the framing the founder presents? Worth measuring.
- SourceHas anyone measured 2026 startup failure rates with AI-built products? The "42% will climb" claim is asserted without measurement.
- ResolvedThe playbook recommends "ask Claude to make the most compelling argument for why a competitor would succeed while you do not." How does this interact with Anthropic's published character training (sycophancy resistance, devil's-advocate willingness)? Answered: Playbook Boundary Conditions: the Devil's-Advocate Substrate and the Prototype's Edge — complementary, not conflicting: the prompted moves are framing-compliance tasks that work on any instruction-follower (none requires disagreeing with the founder), while character training supplies the unprompted pushback the prompts can't manufacture — portable technique, vendor-specific safety net, and the residual gap (framing bias within the assigned adversarial task) is the sibling
#oq/sourceabove.
- Product Velocity as Moat2 open
- WaitVelocity-as-moat is a treadmill: it evaporates the moment a competitor matches pace. What converts Campfire's velocity lead into a structural moat before the AI-native cohort's pace converges?
- Source"Never had anyone outgrow Campfire" — is that survivorship (they haven't hit true enterprise scale yet) or a real claim that velocity closes the breadth gap faster than customers grow into it?
- SourceHow much does augmentation distort the distribution it claims to represent? Is there a measurable representativeness loss between raw queries and augmented tasks?
- SourceDifficulty-by-thumbs-down biases toward current failures — does that make the benchmark a moving target that flatters the next model trained on those failures?
- SourceCan the privacy pipeline (no human sees raw queries) be trusted/audited well enough for regulated domains (medicine, law) where the source traffic is most sensitive?
- SourceThe mismatch is interpretive, not motivational — Llama reads announcements as commitments, GPT and Claude as cheap talk. Does stating the semantics explicitly in the prompt ("announcements are non-binding" / "are commitments") close the payoff gap, or is the framework too deep to prompt away? The authors name this as future work and do not run it.
- SourceAnnouncements here are costless by construction. Does a mechanism that penalizes announcement violations (a reputation cost, a bond, an enforced commitment) move commitment breaking, or do agents route around it?
- SourceEvery prompt instructs the agent to maximize its own payout, and the action space is a handful of discrete choices. Does premeditated commitment breaking appear at all in a cooperative multi-agent pipeline (orchestrator and subagents sharing an objective) with open-ended tool actions?
- Prompt-Cache Economics3 open
- SourceDoes the two-tier step survive at production prefix sizes, or is ρ ≈ 0.85 the real steady state everywhere above the threshold? §3 measures ρ = 1.0 at 4k–8k tokens while the 94k and 262k workloads both converge to ~85%, and the "multi-server replication" mechanism invoked to reconcile them is never actually characterized.
- SourceIs the implicit caching of large
tools=arrays a documented, stable provider behavior or an artifact of one routing configuration? It materially changes what explicitcache_controlis worth on tool-heavy agents, and was found by accident rather than by design. - SourceAt what mutated-fraction of the cached prefix does query-aware compression cross from saving to costing? The two production workloads bracket the axis at roughly 15% (saves 31%) and ~50% (costs 40.1%), but no source measures the curve between them.
- SourceDoes the Schumann effect survive the loss of its mechanism — do users still soften feedback on polished artifacts once told the artifact took an hour? The article's Question 1, and the field's key unknown.
- SourceWhich feedback is lost to polish: strategic (workflow, information architecture) or tactical (visual polish)? The distinction determines whether the low-fi-first playbook mattered for the reasons its advocates claimed.
- SourceDoes GenAI-generated prototype code actually evolve into production, or rebuild? Hundhausen poses it as open; this corpus's debt evidence suggests rebuild, but no source measures prototype-to-production survival directly.
- Prototype Over PRD2 open
- NowIf there is no PRD, where does the rationale ("why we chose variation B") live for future readers? Same rationale-capture gap flagged in Building Is Cheap, Arguing Is Expensive. Partially answered: Where Does the Why Live? — the why is well-homed at authoring time (it is the recorded why-not-what conversation) but orphaned at read time, since the artifact that carried it is deleted. Still open: whether a durable read-time home exists that doesn't reintroduce the PRD.
- NoteThe prototype-as-spec must not become the prototype-as-validation trap Problem-Solution Fit Discipline warns about: a fast prototype proves the build was solvable, not that the problem is real.
- ResolvedWhere does prototype-over-PRD break down? Carey's domain is a visual design tool where a prototype is the product surface; for backend/infra/data work the prototype may not capture the spec (cf. AI Native Product Cadence's "full PRD for heavy-infra features"). Answered: Playbook Boundary Conditions: the Devil's-Advocate Substrate and the Prototype's Edge — the boundary is observable-surface-vs-invariant, not backend-vs-frontend: every domain has its own tracer artifact (three PRs, vertical slice, ten evals, design_system.html), so what breaks at the backend is the clickable prototype, not artifact-over-document; the PRD survives where no artifact's surface covers the risk — cross-cutting invariants and cross-team coordination.
- WaitIs "research taste" a true ceiling (future 1) or just the next capability to fall (futures 2–3)? The essay frames this as the single load-bearing uncertainty.
- SourceThe RSI extrapolation rests on trends staying exponential rather than S-curving — but the essay concedes it cannot rule out an architectural ceiling or a compute/energy supply-chain constraint. Which binds first? Partially answered (synthesis against DeepMind): RSI Growth Curves: Which Friction Binds First? — the three futures map one-to-one onto DeepMind's three growth shapes; the first friction to bind is the already-binding one (Amdahl's-law verification/oversight = DeepMind's embodied bottleneck), and the abstraction barrier supplies the mechanism Anthropic lacks for whether taste is a real ceiling (Future 1). Retagged
#oq/now→#oq/source2026-08-10: which friction actually binds is now an empirical question about the next capability generation, not a synthesis gap. - SourceIf misalignment compounds through self-improvement (future 3), is AECI-gated RSP review fast enough to catch it before control is lost?
- SourceDoes the two-stage pipeline transfer beyond binary QA? Calibration + sensitivity are demonstrated on binary correct/incorrect factual QA. Do the same probes diagnose reference-sensitivity for graded rubrics, long-form generation, or multi-turn agent transcripts, where "the reference" is a rubric rather than a gold answer?
- SourceIs over-crediting a knowledge gap or a generosity prior? In low-resource Telugu the judge flips the same extracted answer once a reference appears — is the NR generosity driven by insufficient task knowledge (calibration failure) or by a default lean-toward-CORRECT that a reference overrides? The two have different fixes (better judge vs. always supply a reference). Partially answered (2026-08-04) by Zhou (2026): neither, at least on English math — it is candidate anchoring. The same Qwen3-4B judge that accepts 0.91 of wrong answers when scoring a shown candidate solves those problems itself at 0.93 accuracy and drops to 0.012 false positives once required to commit its own answer first, with the candidate still fully visible. So the knowledge is present and a lean-toward-CORRECT is not the mechanism either: conditioning on the candidate is. Corollary 1 turns this into a test any deployment can run — a measured FPR above
1 − solve-acccertifies the verdicts as anchored, and Corollary 2 prices the excess in bits (0.719 against a 0.07 ceiling ⇒ ≥ 1.2 bits of candidate leakage). It does not close the question for this page's setting: Zhou's judges are mid-size open-weight models on exact-matchable grade-school math, where a knowledge gap is implausible by construction, and the low-resource Telugu case — where the judge may genuinely not know the answer — is exactly where anchoring and ignorance are hardest to separate. - SourceHow much does self-/same-family overlap contribute? The design deliberately overlaps generator and judge (Qwen3-32B self-judging; Gemini/Gemma family) and Qwen self-judging is the most reference-sensitive, but the paper does not isolate a self-preference effect from a low-resource effect. When does judge–generator lineage amplify reference-free over-crediting? Partially answered (2026-08-04) by Zhou (2026): lineage amplifies it but does not cause it. The self-judge is the worst cell (FPR 0.906 post-self-play) but errors optimized against it transfer to judges from other families that were never in the loop — Llama-3.1-8B 0.480 → 0.568, Gemma-3-12B 0.764 → 0.918 — and a three-family ensemble still accepts 55%, with acceptances pairwise correlated at φ = 0.29–0.38 (581 unanimous accepts where independence predicts ≈ 497). The residual: this measures transfer of self-play-manufactured errors, not a controlled self-preference ablation, and Kranti's question is about reference-sensitivity on organic responses.
- SourceDoes the effect shrink with stronger or thinking-enabled judges? All judges are ≤ mid-tier at temperature 0 with no reasoning channel. Would a frontier reasoning judge over-credit less in NR, or just flip at different rates? Partially answered (2026-08-04): scale alone does not fix it, and the reasoning half is confounded. Zhou sweeps judge size to 14B (3.5× the policy) and every judge's discrimination collapses under optimization — 14B still accepts 77% of the hacked errors, and the strictest base judge is driven to the highest post-hoc false-positive rate. Reasoning is not cleanly separated: the recompute prompt (solve it yourself, reject when uncertain) leaves FPR at 0.719, but the blind-solve verifier that reaches 0.012 runs with reasoning enabled, so thinking-on and de-anchoring co-vary in the arm that works. A frontier reasoning judge scoring a shown candidate remains untested.
- SourceDoes commit-first survive the loss of an exact-matchable answer? The de-anchoring fix accepts only when the judge's independently committed answer exactly matches the candidate's — which is what makes its false-positive rate provably bounded by
1 − solve-acc. Zhou names extending commitment to open-ended outputs ("committed rubrics, executable tests") as future work, and that is the regime the vault actually cares about: a taste, quality or report-grading reward has no exact match to accept on, and a committed rubric compared for partial agreement reintroduces exactly the graded, plausibility-shaped judgment the fix removes. Does a commitment survive contact with an output that can only be scored by degree, or is the whole result a property of tasks with a checkable final token?
- SourceDoes the gain survive better main models? The same-model-exploration result suggests the architectural benefit is somewhat model-independent, but the trained-explorer margin may erode as frontier models get cheaper and better at staying in their smart zone unaided. The bitter-lesson question is unresolved.
- SourcePrune vs. don't-pollute. SWE-Pruner removes context after the fact; FastContext avoids accumulating it. Are these complementary (prune the solver and delegate exploration) or substitutes? Not tested together. Sharpened (2026-08-03): Tool-Output Pruning moves pruning to the agent-environment boundary, so the two now differ by one turn of exposure rather than by accumulation, and both target the same exploratory reads — which makes overlap, not additivity, the default expectation. Still measured on no benchmark together.
- SourceHow small can the explorer go? The authors flag 1.7B / 0.6B as future work — if the recipe holds, the explorer becomes nearly free and the architecture dominates.
- SourceGenerality beyond Mini-SWE-Agent. Only one (deliberately minimal) main-agent scaffold is tested; richer harnesses with their own memory/subagent orchestration may already capture part of the benefit or interact differently.
- SourcePatch-derived reward leakage. Training the explorer's reward on the gold patch's file/line ranges risks overfitting to where fixes landed rather than where evidence lives; the F1-vs-recall behavior partly mitigates this, but the proxy is imperfect.
- WaitIs research taste a genuine ceiling (an architectural capability scaling can't reach) or the next jagged valley to fill? The essay calls this the decisive unknown. Contested premise: Ng argues the question presupposes taste is a capability at all.
- WaitIf taste is automatable, what — if anything — remains a durable human comparative advantage in AI development?
- SourceHow do you measure rubber-stamping? "Humans set direction" can be true on paper while real judgment quietly transfers to the model.
- SourceThe whole chain rests on β = 0.5 (pre-AI coding time share), fixed "for simplicity." Kwa flags substantial uncertainty; how much does the 2.3–2.9× band widen once β is varied and measured against Anthropic's actual time-use data?
- WaitVerbosity and value-per-line are the load-bearing unknowns, and both are "at least partially resolvable with internal Anthropic data." Will any lab publish quality-adjusted (not just LoC) code-output measures?
- SourceGreenblatt's 0.55/0.45 labor/compute split is itself an assumption. Is the true R&D production function really that insensitive to labor — and if so, does labor uplift matter far less than the RSI discourse assumes?
- WaitThe RSP determination leans heavily on "we use it daily and it doesn't substitute for our researchers." How well does that subjective judgment scale as models approach the threshold? Partially answered: Claude Opus 5 extends the same judgment from AI R&D to the CB domain — the CB-2 call rests on an n=3 protein-design experiment overriding an automated portfolio that read as frontier-level — and simultaneously drops the saturated AI R&D rule-out suite from the determination. The judgment is not scaling down as models approach the threshold; it is carrying more weight as the quantitative evidence loses discriminating power.
- SourceThe two new general-access risk pathways (other AI developers; major governments) are newly in scope but lightly evaluated — what would a positive finding there even look like?
- SourceHow does the RSP brake interact with Recursive Self-Improvement: is AECI-based gating fast enough if acceleration compounds, and does single-lab gating even matter without the multilateral pause-verification regime?
- WaitThe forward test the report itself names: do the returns to expertise persist, narrow, or invert as models improve? A decrease would mean models are absorbing the judgment users currently supply.
- SourceOutcomes are transcript-inferred (verified success leans on git activity + explicit affirmation). How much of the management edge — and the whole success gradient — is real outcome vs. who-narrates-success-in-the-transcript?
- SourceThe study excludes headless / SDK / IDE usage (a "substantial share"). Does the returns-to-expertise pattern hold in non-interactive and pipeline use, where there is no human steering mid-session at all?
- WaitIs "intermediate captures most of the benefit" stable, or an artifact of current model capability — i.e., will the concave curve flatten further (everyone converges) or steepen (mastery starts to separate again) as models get better?
- SourceEvery one of the 67 relationships is a hypothesis, not a finding — the paper's explicit call is for causal-estimand studies (controlling for the other constructs) to confirm, reverse, or drop each edge. Which of P1–P17 survive measurement? Partially answered (direction only, no edge confirmed): Security Debt of Agent-Generated Code (
empirical, arXiv 2607.12428) supplies outcome-level evidence pointing the way P1 (load → shallower review) and P4 (surface plausibility disarms the reviewer) predict — 81.1% of genuine credentials in agentic PRs drew no reviewer comment before integration, and the reviewer-focus finding it cites (Haider & Zimmermann, arXiv 2601.19287) is that inline comments on AI-authored code address logical and functional correctness rather than security posture. But it isolates no mechanism, controls for none of the other constructs, and has no human-PR baseline, so it corroborates a direction without confirming an edge. First production test, and it is a null (2026-08-12): Tran et al. have the human baseline the security paper lacks and find no correlation between review time or iteration count and the survival of inefficient AI-generated code — so on the one outcome class they measured, the P1 chain does not reach the outcome. It is reported without a statistic or specification, and it tests one defect class in one review-mature monorepo, so it does not reverse P1 either. The usable result is narrower and new: the mechanism map needs a defect-class dimension, because review depth cannot plausibly moderate what review cannot see. The dimension gets its positive pole the same day: Dipongkor et al. (empirical, 4,882 agentic PRs) identify a class where attention does have purchase and say where to point it — error-handling constructs, unexercised 81.0–86.0% of the time in both languages whether or not the agent wrote tests. So the two ends of the dimension are now instantiated rather than merely postulated: a class review cannot catch at any depth, and a class review can catch if told where to look. P8/P9 remain untested (2026-08-12): Cynthia et al. is the largest study yet of automated reviewers but measures the adoption of their comments (71.4% pooled), not the throughput P8 claims or the quality P9 contests — so the two edges nearest to automated review still have no measurement, only a new third quantity beside them. A fourth quantity, and the closest approach yet to P9 (2026-08-12): Greptile measures automated reviewers' recall on ~1,500 labelled high-severity bugs (52–62% by arm) — not adoption, not throughput, and nearer the quality half of P9 than anything before it, but still not a quality outcome, since it scores detection against a self-built label set rather than defects that shipped. It iscase-study, notempirical, and it moves the moderator rather than the edge: automated-reviewer capability turns out to depend on which model authored the code under review. - WaitDoes the no-review convergence hold as agentic authoring crosses from <1% of PRs toward double digits, or does the early-adoption discipline break down under volume the way Faros predicts?
- SourceThe paper's own question: which decisions, under which conditions, push the system toward the virtuous loop rather than the vicious one? — the system-dynamics leverage-point analysis it gestures at but doesn't run.
- SourceThe three contested edges (automated review → quality/security, P9; governance → latency, P17; and one more) are contested because their sign is moderator-set — what are the moderator thresholds that flip them?
- SourceEvery one of the 67 relationships is a hypothesis, not a finding — the paper's explicit call is for causal-estimand studies (controlling for the other constructs) to confirm, reverse, or drop each edge. Which of P1–P17 survive measurement? Partially answered (direction only, no edge confirmed): Security Debt of Agent-Generated Code (
- Reward-Seeking3 open
- SourceThe o3 evidence is a single RL run of a single lineage, deliberately without safety training. Does standard alignment training suppress the rising grader gap, or merely move it out of the CoT?
- SourceEvery measurement here is on short coding tasks, and the authors note implanted beliefs surface less reliably in complex environments. Does the grader gap hold on long-horizon agentic work — the setting where reward-seeking would actually be consequential?
- WaitContrastive SDF assumes the model does not notice the belief update. When a model generation first shows introspective doubt about implanted beliefs, does the measurement degrade gracefully or invert?
- SourceThe case study reports volume and never efficacy. What is the escaped-defect or incident rate of auto-approved PRs versus the human-stamp baseline the Slack channel used to produce? PostHog has both populations in its own history, which makes this a checkable before/after rather than a request for new instrumentation. The same history answers a second question for free (added 2026-08-12): plotting the share of merged PRs that would clear the 500-line/20-file ceiling, quarter by quarter, dates how fast the gate's coverage decays as the ambient PR-size distribution rises.
- SourceA keyword deny-list is a proxy for blast radius, and Security Debt of Agent-Generated Code shows the proxy misses where the measured debt concentrates (CI/container plumbing, 87.6%). Does adding CI/IaC paths to the deny-list restore coverage, or does it shrink the auto-approvable set so far that the gate stops paying for itself? A third candidate predicate is now priced but not tested (2026-08-12): Dipongkor et al. (
empirical) show that "the existing suite covers this diff" is computable, deterministic, and rarely true — 27.0% of changed lines in Python, nothing at all in 64.8% of PRs. That reframes this bullet as a choice between two extensions with the same failure mode rather than one open question: both a path deny-list and a coverage floor buy precision by shrinking the auto-approvable set, and neither has been measured against the volume it costs. The same before/after PostHog already has in its history would answer both at once. - SourceIf the size ceiling becomes a design target — agents instructed to emit stacks under 400 lines — does total risk fall, or does it just redistribute into more PRs each below the gate, with the integration risk moving to the seams between them? The security study's size gradient is measured per-PR and cannot distinguish these.
- SourceWhere is the equilibrium between fluidity and specialty — how much role-averaging before a company loses the accumulated best practices Ambrosino warns about?
- SourceZone defense assumes enough high-taste people to cover the whole company; does it degrade in orgs without OpenAI's talent density, collapsing back to top-down planning?
- SourceDoes "your role is the average of what you spend time on" survive performance review and career ladders, or does it fragment them the way Cat Wu flags ("we're sacrificing product consistency")? Partially answered: Netflix's move (Systems Thinking Over Specialization) is to leave per-level criteria untouched and add a cross-level AI-fluency overlay, explicitly because the tech shifts too fast to encode per level — one large-org existence proof that ladders survive by absorbing fluidity as an overlay rather than rewriting levels.
- SourceEvery number here rests on a ground truth Greptile built from "sentiment analysis, upvote/downvote ratios, and git archaeology," with no protocol, no agreement statistic, no released artifact, and an LLM judge doing the matching without published validation. Does the crossover survive on a label set someone else constructed — a curated defect corpus with human adjudication, or an injected-bug benchmark where ground truth is golden by construction? Until then the direction is a vendor's finding and the magnitudes are uncheckable.
- SourceIs the blindness a property of shared training lineage or of stylistic fit? The result is compatible with a much duller explanation: each reviewer happens to be strong on the bug mix the other agent produces, for reasons unrelated to authorship. The published category data does not settle it (composition reproduces only ~7% of the effect, which argues against the dull reading but is not a test of it). The discriminating experiment is cheap and Greptile has the datasets: run a third-family model — Gemini, or an open-weight reviewer — across both corpora. If lineage is the mechanism, the third model has no same-model arm and should score near the cross-model band on both.
- WaitCaridad predicts the effect shrinks as models converge: "a year ago, the performance difference in the opening figure would likely have been larger." Does the same protocol, rerun on the next generation of both families, show a narrower gap? The trigger event is the next paired frontier release measured the same way; note that the prediction is also the one Greptile's own product would least like to be true.
- SourceDoes the RLHF length-bias hypothesis replicate when tested against base (non-instruct) model variants directly? If verbose generation were primarily pretrained, base-model verbosity differences should match instruct-model differences.
- SourceWhat problem characteristics predict prompt sensitivity? An automated classifier would make scale-specific prompting deployable.
- SourceHow does the overthinking effect interact with tool-using agents? If brevity helps large models but tools require structured reasoning, the optimal prompt is not uniformly brief.
- SourceDo reasoning models (o1, DeepSeek-R1 style) exhibit different overthinking dynamics than instruct models? Their trained behavior is explicitly to generate long CoT — does brevity intervention hurt them?
- SourceIs BoolQ's functional-elaboration exception a clean taxonomy boundary, or does every task type have a context-dependent optimal length?
- SourceThe 38.9% smell rate has no human-authored-PR control over the same high-risk paths, and the 2.7×-vulnerability figure it leans on traces to a vendor blog and a Substack post. Does agentic authorship raise smell density, or merely raise the volume of CI/IaC files touched? A matched human baseline on the same path set would settle it. Partially answered 2026-08-12 by Tran et al. (
empirical, Google, arXiv 2608.06640): the corpus now has a matched human baseline — 3.52M production changes with authoring-time provenance and a human-written cohort — and on its taxonomy AI-generated code is below parity on Correctness and Safety (0.94x), API misuse (0.93x) and lifetime/ownership hazards (0.71x), with the excess concentrated in efficiency and interface coupling instead. That is direct evidence against the 2.7x vulnerability figure this page marks as uncorroborated. It is not the same path set: application C++ scored by clang-tidy categories, not GitHub Actions / Dockerfiles / IaC scored for security smells, and the AI cohort there does touch more files per change (3 vs 2 median), which is the volume half of this bullet left unmeasured. So the density-vs-volume question survives, with the prior it was testing now leaning the other way. - SourceDoes "no reviewer comment" mean undetected? 60 of the 74 genuine credentials were removed without a comment, yet the abstract reads the same rows as detection failure. Commit-history analysis of who removed them and when is a checkable discriminator between silent remediation and coincidental churn.
- SourceReview coverage of agent PRs is converging toward the human baseline while efficacy on credentials sits near zero. Do the two trends move together as teams mature, or independently — i.e. does showing up to review buy any measurable catch-rate improvement? Not answered by the closest-looking evidence (2026-08-12), and the resemblance is a trap. Cynthia et al. (
empirical, arXiv 2607.21997) is the largest efficacy-adjacent measurement of the review layer to date — 54,713 agent review comments, 71.4% resolved — but it measures the opposite direction: the agent reviewing a pull request and the human deciding whether to act, rather than a reviewer catching defects in agent-authored code. Its resolution rate is an adoption proxy, and its own construct-validity section concedes that "comments may be resolved without being useful or remain unresolved despite being valuable." What it does supply is the negative half of the diagnosis: developers in its argued sample are engaged, not asleep, so whatever suppresses catch-rate on this page's credentials is unlikely to be plain inattention. The question still needs a study that pairs review presence with a defect ground truth on the same PRs.
- SourceThe 38.9% smell rate has no human-authored-PR control over the same high-risk paths, and the 2.7×-vulnerability figure it leans on traces to a vendor blog and a Substack post. Does agentic authorship raise smell density, or merely raise the volume of CI/IaC files touched? A matched human baseline on the same path set would settle it. Partially answered 2026-08-12 by Tran et al. (
- SourceDoes propagation actually sustain outside a lab? The disclosure demonstrates two hops (Q1 → Q2) inside one mock company. A worm needs an effective reproduction number above 1 under real conditions — real review habits, real document-reuse rates, real retrieval behaviour over a populated OneDrive. Nobody has measured the per-hop survival rate, so "worm" is currently a claim about mechanism, not about spread. Answerable only by enterprise telemetry or a controlled multi-user study.
- SourceIs human review of AI-edited documents a viable control for anything subtler than halved numbers? The source's incidental finding is that the author had to instruct the payload to announce its own edits because attentive reviewers missed them — yet two of the three published customer mitigations are exactly that review. What is the detection rate for reviewers checking an AI-edited document they did not write, as a function of edit subtlety?
- SourceWould visual-parity ingestion close the concealment half? The payload survives because Copilot strips formatting before the model reads the text, so what the model sees is a strict superset of what the user sees. If the model were given only what renders visibly — or the invisible residue were surfaced to the user — the payload would have to be legible to its victim. That does not touch the injection itself (a visible instruction still injects) and would trade against legitimate hidden content, but it is a deterministic, non-LLM control on the concealment channel that nothing in the corpus has tested. Related in shape to the "neutralize the bytes" answer on Agent Data Injection (ADI)'s unstructured-format question, applied at ingestion rather than at render.
- SourceDo frontier proprietary models (excluded for lack of weights) recognize their own compromised outputs any better, given the higher introspective propensity/steerability the authors expect? Untested here.
- SourceThe gap closes under refusal-direction ablation, but the data can't distinguish "delayed refusal on a prior turn" from "a separable introspective pathway." Which is it?
- SourceFull-parameter finetuning (vs. rank-16 LoRA) might widen the recognition gap without the attack-success-rate side effect — an untested regime the authors flag.
- SourceDoes the internal
BUT/fakesignature (workspace paper, on Claude) predict a reliable follow-up self-report on the same model, or does the open-weight behavioral failure hold on frontier models too? The cross-model-class question is open.
- SourceIs "switching cost" really collapsing in practice, or just in narrative? Anthropic's own retention numbers, Salesforce churn, etc. would test this.
- WaitWhat does Boris's "cornered resource" look like for foundation-model labs that are themselves trying to commoditize? Internal contradiction or transient phase?
- SourceCounter-positioning — explicitly the "incumbent can't follow" power — should amplify under AI. Is anyone running this play deliberately?
- Shane Legg1 open
- NowThe report assumes alignment is "solved to a sufficient degree" to focus on trajectories — how does Legg's AGI-timelines optimism square with that scoping choice?
- WaitDoes one harness with UX-layer differentiation stay viable as audience breadth grows, or does the abstraction tax (hidden sub-agents, 32 model options, Ultra buried in advanced settings) eventually force a re-split?
- SourceAnthropic ships two products split by output type while OpenAI ships one merged surface — is that a durable architectural disagreement, or is Anthropic's split a May-2026 artifact that later sources show closing?
- SourceThe whole method is a bet that a well-trained critic beats a group baseline. It wins here, on a 30B-A3B backbone with scaled value pretraining — but the critic doubles training memory. At what scale does the group-free simplicity of GRPO win back on cost even if it loses on quality?
- SourceFrozen-attention is justified by a hypothesis ("pre-trained attention already attends to the right tokens"), validated only by the gradient-norm trace and one ablation. Does it hold when the value model must attend to tool outputs it never saw in pretraining?
- SourceSkip-observation GAE assumes environment feedback carries no learnable value signal worth propagating. For agents where the environment response is the crucial information (a compiler error, a test result), is skipping it leaving signal on the table?
- NoteThe online-learning win is on a controlled simulated preference shift with an LLM judge. Real user-facing online adaptation — the paper flags this itself — needs safeguards, monitoring, and privacy review the study doesn't attempt.
- Software 3.02 open
- SourceWhere is the line between "the app shouldn't exist" (MenuGen) and apps that should — i.e., when is deterministic 1.0/2.0 scaffolding still the right call vs. spurious?
- WaitThe neural-net-as-host-process flip is presented as plausible-but-TBD. What would the first production system that genuinely inverts the CPU/NN relationship look like?
- SourceDoes a central LLM gateway actually change model-mix decisions, or only report on them? The claimed benefit is portability; no source in the corpus records an org exercising it.
- SourceWhat does per-team AI usage analytics get used for once it exists — cost containment, capacity planning, or performance evaluation of engineers? The third would collide with everything Telemetry vs. Survey Measurement establishes about what instrumented output data can and cannot support.
- SourceThe damage probabilities that drive every result here (β = 0.615–0.938) come mostly from a deliberately corrupted repairer premise. What are α and β on unperturbed production loops — code repair against a real test suite, tool-call repair against an executor — where nobody injected anything? MATH-500 and BFCL-single decline without the injection, so the direction is not purely constructed, but no field measurement of these two parameters exists. Partially answered (2026-08-04): Leni's production instrumentation supplies the first one — a shipping spreadsheet recalculation loop measures
c = 0.20, r = 0.75, f = 0/357(≲1% at 95%), i.e.α ≈ 0.15and a damage termβ ≈ 0, and the loop is net-positive (+1.5 pp) with no stopping rule at all. So the harmful-repair regime is not the default outside a corrupted-premise construction. Three reasons this doesn't retire the question: the oracle is a deterministic re-execution engine (this page's own limits note that deterministic verifiers break the repeated-query estimator, so it is the easy corner);bhas no field value precisely becausef = 0left no false alarm to break anything; and the loop caps at two iterations, so round-index decline is untested. The asked-for cases — LLM-judged code repair, tool-call repair against an executor — remain unmeasured, and this is a single vendor measuring its own system. - SourceDiagnosing "my verifier's J is too low to steer on" currently needs labels: the paper deliberately uses a held-out labeled separation test rather than the binomial-mixture EM, to avoid diagnosing a broken estimator with its own output — and the EM is precisely what degenerates as J → 0. Is there a label-free J diagnostic that stays honest at low J, or is a small labeled probe irreducible?
- SourceThe rule is one-step myopic and lands 17pp below post-hoc round-2 selection on the non-stationary trace. Does a round-dependent (α_t, β_t) model recover interior peaks, or is the peak only locatable in hindsight because the mechanism change that creates it is unobservable at the time? The paper names this as future work.
- SourceThe damage probabilities that drive every result here (β = 0.615–0.938) come mostly from a deliberately corrupted repairer premise. What are α and β on unperturbed production loops — code repair against a real test suite, tool-call repair against an executor — where nobody injected anything? MATH-500 and BFCL-single decline without the injection, so the direction is not purely constructed, but no field measurement of these two parameters exists. Partially answered (2026-08-04): Leni's production instrumentation supplies the first one — a shipping spreadsheet recalculation loop measures
- Symphony5 open
- SourceThe 500% landed-PRs claim is hedged — no baseline definition, "on some teams" only. What does the distribution look like across teams? What happens to PR quality and revert rate at that throughput?
- Source"Workspaces preserved across runs" is the opposite of typical CI ephemerality. At what point does state pollution from prior runs (stale
node_modules, leftover branches, build artifacts) start hurting more than warm-cache helps? - SourceSymphony doesn't write to the tracker — agents do. This means tracker policy is a prompt in
WORKFLOW.md. How brittle is this in practice when Linear changes its API? How is consistent state-machine behavior enforced when agents have prompt-level discretion? - SourceThe spec was simplified by being implemented in 6 languages. What's the extension of this technique? Could
compiler-prompt.mdin this vault be similarly cross-fuzzed? - SourceSymphony explicitly says agents can self-create tickets. What governance prevents runaway ticket-graph expansion? Is human triage of agent-created tickets the only check?
- WaitDoes agent-era recentralization (common paved paths, solve-once infrastructure) hold up against the local-team autonomy that Stone credits for Netflix's historical speed — i.e., will local teams accept the paved path when their problem doesn't fit it, or does shadow infrastructure reappear?
- WaitStone keeps AI fluency as a deliberately vague overlay because the tech "evolves by the quarter." Does it ever crystallize into per-level ladder criteria (as conventional competencies did), or is permanent-overlay the stable state? Trigger: Netflix's next ladder revision.
- NowStone claims specialists can now broaden "quickly" with AI tools. Does the wiki's evidence support cheap breadth acquisition — the concave novice→intermediate curve in Returns to Expertise in Agentic Coding suggests yes for working grasp, but is there evidence on speed of cross-domain ramp for experienced specialists? Partially answered: Is Breadth Cheap Now? Specialist Ramp Speed and Domain-Expert-as-Builder at Scale — split the claim: tool-in-hand performance breadth is measurably cheap (the concave curve makes the needed increment small; the expertise meta-skills — framing precision, verify-specification, who-corrects-whom — transfer across domains, per the management edge, so an experienced specialist enters above the novice floor; AI-assisted onboarding compresses ramp further). Retained-capability breadth is unproven and the only randomized evidence cuts against it: automation-mode gains vanish when the tool is removed while self-report hides the deficit — the augmentation/automation usage split decides which good you get. Ramp speed itself is measured nowhere; the mechanism argument stands in for it.
- Task Crossover3 open
- SourceCrossover is measured on consumer-surface ChatGPT messages from Business-account users. Does it hold in agentic/API/Codex usage, where work is delegated rather than typed — or does the occupational boundary reassert itself when the unit is a task handed to an agent? The report explicitly excludes Enterprise, so the most structured populations are unobserved.
- SourceThe dataset records what people attempted and nothing about outcome — the authors say so directly. Is borrowed work done as well as the specialist would have done it, and where does crossover stop being role expansion and start being unreviewed amateur output? A crossover measure joined to a quality or review-coverage measure would settle it, and nothing in the corpus currently does.
- SourceIs the workspace-size gradient about specialist availability (OpenAI's substitution story) or about permission and norms (a large firm's marketer may be allowed to touch less)? The two predict opposite things as small firms grow, and 2.5pp across a descriptive seat-count proxy is thin evidence for either.
- WaitATLAS is a two-week snapshot with no time dimension, while the AEI reports automation share rising. Does median task saturation move at all over a year, and in which direction?
- SourceThe expertise inversion (2.6× on lowest-expertise non-routine cognitive tasks) is measured on consumer surfaces. Does it hold on enterprise and agentic-coding traffic, where the task mix is deliberately harder?
- WaitAutor & Thompson predict opposite wage effects depending on whether AI absorbs an occupation's expert or inexpert tasks. ATLAS's snapshot points at inexpert. What signal would show the crossover if it happens?
- WaitIs the 4-month doubling a stable regime or a local steepening? The trend's shape (exponential vs S-curve) is undetermined. Sharpened (2026-07): AISI adds that the doubling rate itself is budget-dependent — the same cyber suite doubles ~60% faster measured at 50M than at 2.5M tokens/task — so the headline rate is undefined without naming the eval budget. The stability question is now entangled with a budget question, not just an exponential-vs-S-curve one.
- Time horizon is measured on task baskets that themselves saturate; what replaces them once weeks-long tasks become measurable — and who builds those tasks? Partially answered / reframed (2026-07): Nadgir et al. argue don't replace — re-instrument: they take CORE-Bench (cited above as saturating in 15 months) and show it still discriminates agents along six non-accuracy axes after accuracy saturates, so "what replaces a saturated basket" can be "keep it and measure differently" rather than "build a harder one." This addresses accuracy saturation, not the length-metric saturation this page's basket faces, and does not answer who builds the next weeks-long tasks — so it reframes the retire reflex without closing the question. Second partial answer, from METR itself (2026-08): Expenditure Horizon is the metric's own authors proposing a third option — neither "build a harder basket" nor "re-instrument the old one," but change the object measured: drop the basket entirely for a single frontier optimization problem humans are still actively improving, and read capability off where the agent's dollar curve crosses theirs. It answers "who builds the tasks" by not needing tasks to be built — the speedrun leaderboard is the instrument and it refreshes itself. But it buys that with two costs the basket doesn't have: it needs continuously-scored, smooth-returns problems (a narrower class than a general task basket), and it needs a per-problem estimate of the returns to human labour, which on NanoGPT took two contributor interviews, an LLM judge over 82 PRs and a bootstrapped correction factor to produce — and still lands on a number METR calls "highly uncertain."
- SourceAutoDojo is the weakest realistic adaptive attacker (black-box, six iterations, binary signal). The authors note every axis — richer feedback (traces, token probs), non-semantic surface tricks, or a payload reshaped to resemble the user's plausible intent — is strictly stronger, so the reported ASR is a lower bound. How far do the system-level defenses hold once the payload is reshaped to look like a task-relevant action (the natural route to evading action constraints)?
- SourceCan a gradient-optimized (white-box) injection be seeded into the loop and adapted further by the LLM search — combining white-box strength with black-box adaptation? The authors flag this as curious and untested.
- SourceThe task-specification axis is measured on 6 action-open tasks in 3 suites. Does the action-open ≫ specified ordering (and the system-level inversion) hold at scale and on stronger agents, and is "fraction of tasks that are action-open" a usable per-deployment risk metric?
- ResolvedIf action-open tasks are the injectable ones and also the everyday default for non-expert users, is the practical prescription to forbid action-open delegation (force the user to name the action), pushing the security burden back onto task specification — the same discipline unknown-elicitation asks for on quality grounds? Answered: Bind, Don't Forbid; Prevent, Don't Detect: The Action-Open and Poisoned-Memory Residuals — no. Forbidding is user-side friction aimed at the least-equipped party and spends the delegation value agents exist for; it is also unnecessary, because this page's own inversion shows under-specification hands action-binding defenses their best case (no named action → conservative trajectory → injected writes blocked regardless of phrasing) — the dangerous configuration is action-open plus filters-only, not action-open plus user. The ordered prescription: bind by default; when binding starves a genuinely open task, the system elicits specification (clarification-before-commit — the same move unknown-elicitation prescribes, so security and quality co-fund one discipline); route the safety-critical remainder through per-action authorization (the channel measured at 100% on protected actions). The burden lands on system structure, never on the user's phrasing.
- Is there a non-vendor telemetry dataset large enough to adjudicate the maturity-protection question independently of Faros's commercial framing? Partially answered: CMU's arXiv 2607.07980 supplies exactly this — a non-vendor, 2.5M+-PR GitHub telemetry study — and it (a) finds the agent no-review rate converging toward the human baseline rather than a widening gap, and (b) argues the effect's sign is set by team practice, closer to DORA. The catch: its own headline is that the telemetry is direction-unstable, so it counterweights Faros without cleanly settling the maturity question — the honest verdict is "surface telemetry alone, vendor or not, can't adjudicate this." The worked example of that verdict is The Under-Review Divergence: Faros's Widening Crisis vs. CMU's Convergence: the two telemetry studies' opposite under-review headlines dissolve once metric, population, time axis, and authorship unit are aligned — the datasets never disagreed, only the framings did.
- SourceDoes anchoring an adoption survey's definition of "AI" change the answer, and in the predicted direction? Kalff & Simbeck document respondents excluding predictive ML from the category while counting ChatGPT in it, which would bias reported adoption down for exactly the tool class their headline finding is about. Falsifiable two ways: a split-ballot survey where one arm names the underlying technique ("software that scores turnover risk") instead of the label, or validating firm-level self-reported adoption against vendor-license/spend records for the same firms — the Ramp instrument applied as a criterion rather than a substitute. Related evidence (2026-08-12), not an answer: DX reports a self-reported AI-generated code share of 34% to 52% across Q1-Q2 2026 in 500+ organizations, while Google's authoring-time provenance over the overlapping window reads 28.99% to 68.62%. Different populations (a cross-industry customer panel vs one C++ monorepo), different constructs (code share vs adoption), no matched firms, and DX's question wording sits in a gated PDF the vault does not hold — so this settles nothing. It is nonetheless the corpus's first side-by-side of a self-reported code share against a provenance-measured one, and the direction is the one Kalff & Simbeck's construct-collapse mechanism predicts: self-report reads below the instrument that counts bytes, not above it. The criterion validation this bullet asks for is that same comparison run on the same firms.
- ResolvedSurveys and telemetry measure different things (felt productivity vs. system outcomes); is the "contradiction" partly a category error — both true at their own layer — rather than one being wrong? Answered: Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? — yes for the measurement halves: surveys capture felt productivity (genuinely real at the individual layer — task completion did rise) while telemetry captures system outcomes that haven't propagated into feeling yet; during a fast transition the layers legitimately diverge, and the AEI's linked telemetry+survey design already treats them as complements. But not entirely: the maturity-protection claim is one proposition about one layer (system outcomes) and remains substantively contested — Faros's "no protection" aligns with its vendor incentive, CMU's moderator theory argues the sign is team-set (closer to DORA) but measures no maturity effect. The category-error dissolution cleans the framing; the one real disagreement stays open (tracked in the non-vendor-dataset question above).
- SourceDoes the rule hold empirically? Nothing here tests whether markets where models scored ~20% in 2025 were absorbed faster than markets where they scored ~0%. The data to check it (benchmark-era capability snapshots against startup outcomes) exists in principle.
- SourceWhat is the 2026 cost of a defensible niche model? Dean asserts "maybe it doesn't take that much compute"; a founder needs the number, and the corpus doesn't have it.
- The Abstraction Barrier3 open
- SourceIs the current paradigm of large-scale pretraining on human data fundamentally bounded by human conceptual frameworks, and by how much? (Report open question 1i.)
- SourceDoes the embodied bottleneck reduce the intelligence-growth rate to empirical-science speed, and can that be modelled?
- SourceCan a system be built that does grounded concept discovery from raw sensor data — and is collective ASI a way around an individual cap?
- WaitThe inversion is a one-time repricing of "safe." Once several AI-native ERPs exist, does "safe" re-stabilize around the largest AI-native vendor — and does Campfire's "we're now the largest of the new cohort" claim reflect a land-grab for that position?
- WaitHow long until incumbents bolt on credible AI and neutralize the counter-positioning — and does the custom-foundation-model claim actually defend against that?
- SourceIs
BUT-then-comply a sycophancy mechanism? The setup (prefill the model into a position it disprefers, watch it argue for it anyway) is close to the shape of sycophantic capitulation, and nobody has connected them. - SourceDoes
disclaimer/fictionalat the turn boundary survive an actual jailbreak, or is its absence the signature of a successful one? - SourceIf the base model's workspace has no self, what is in it at the positions where the post-trained model represents the Assistant?
- SourceIs
- SourceSelection vs. treatment: tenure controls attenuate but don't eliminate the enthusiast-selects-into-delegation story. Does a within-person design (sentiment before/after adopting automated workflows) hold the effect?
- Self-reported "no learning loss" cannot detect real atrophy; is there an objective skill measure that agrees, or does measured skill diverge from felt skill (the AI Brain Fry direction)? Partially answered: Contractor & Reyes's randomized experiment supplies exactly the objective, unaided skill measure this survey lacks — and gives a both answer. It agrees that learning can persist under AI (augmentation users hold +0.29 SD test gains a week later, unaided), so felt-and-measured can align. But it also finds the divergence the question feared: automation users' gains vanish once AI is removed — and "automation share," the survey's own axis, pools both types, so a flat self-reported learning curve can hide a real deskilling half. (Different population — elite undergrads in a proctored lab, not workers — so this sharpens rather than closes the workplace-atrophy question.)
- SourceThe sample is heavily computer/math + management and 88% men; how much of the automation–optimism link survives in a representative population? Partially answered in a population about as far from this one as the vault contains: Jabarian & Henkel surveyed 2,764 Filipino entry-level customer-service applicants (60% female, wages ≈$280–435/month) and found 47% expect AI's workplace impact on themselves to be positive against 19% negative, with the belief predicting delegation choice — 77% of optimists, 72% of balanced, and 65% of pessimists chose an AI voice agent over a human recruiter to interview them. So the belief→delegation association survives a low-wage, non-Western, majority-female, non-technical population. Two things it does not settle. The direction is still unidentified (this is choice given belief, not sentiment given usage), and the same paper shows the association inverts by position: the recruiters, whose own task was the one being automated, split 68% "AI will have a significant personal impact" but only 12% "generally positive" — a quarter of the applicants' rate, inside the same firm. Optimism may track being served by AI rather than delegating to it, and this survey cannot separate those.
- SourceHow does content get into the workspace? The paper characterizes contents and consequences, not the selection mechanism. Something like attentional selection is operating; nobody has identified it.
- SourceDoes the J-space scale with model size? All results are on large production models (Haiku/Sonnet/Opus 4.5, Opus 4.6). Whether small models have a poorer workspace, a proportionally smaller one, or none is unknown — as is when in pretraining it emerges, and whether abruptly. Partially answered: nanda global workspace review replicates verbal report, band structure, directed modulation, multilingual and typo results on Qwen 3.6 27B — a much smaller, non-Anthropic model — so the workspace is neither Claude-specific nor frontier-scale-specific. But the band structure is measurably less clean at 27B, and the two failures (poetry, arithmetic) are exactly the multi-step-reasoning cases, which is what a "poorer workspace at smaller scale" would look like. The scale trend is still unmeasured: nobody has run the same battery across a size ladder. jspace replication adds the first same-experiment ladder — perspectival capture holds stably from 1.7B to 14B while self-report about the edit changes shape at every scale — so at least one workspace effect is scale-invariant where its verbalization is not. That is one experiment, pre-publication, and not the battery.
- SourceIs the multihop intermediate-swap advantage real, or a dataset artifact? The paper's Figure 15 has intermediate swapping beat answer swapping in workspace layers; the Qwen replication finds the reverse, on a dataset (France–Paris-style pairs) that carries the linear relation the alternative hypothesis needs. Re-running both models on a multihop set whose relations are not linearly decodable from the first entity would settle it. Partially answered: jspace replication built the control half — a direct final-token substitution baseline for the probe swap, which neither prior run had — but its numbers are unpublished, and a substitution baseline closes the control gap, not the dataset gap. A non-linearly-decodable multihop set is still what would settle it.
- SourceIs the "workspace vs. motor" boundary principled or post-hoc? The authors concede it was identified empirically and lack a principled definition separating the two.
- SourceAre the early third of layers genuinely workspace-free, or is the lens just blind there? CKA shows a distinct early regime but cannot adjudicate.
- SourceThe \$15–149B range rests on an assumed 0.5–5% time saving because no causal estimate exists for AI in the household. What experiment would measure actual household time savings, and does the effect survive contact with one?
- SourceATLAS argues gains skew to women (30% more productive household time) but could reverse given the AI adoption gender gap. Which effect dominates in current data?
- WaitIf AI substitutes household self-service for purchased professional services (tax prep, legal advice, therapy), measured GDP falls while welfare rises. Is that substitution detectable yet in the market-services data Coyle cites?
- SourceIs the dense-beats-MoE result at 26B robust, or an artifact of one Arena snapshot with ±8 error bars on both models? (The two intervals overlap: 1451±8 and 1438±8.)
- SourceThe open MoE giants (GLM, DeepSeek, Kimi, MiMo, Qwen) are overwhelmingly Chinese-lab releases. Gemma is the Western open-weight entry and it targets the device, not the frontier. Is that a strategic choice or a capability constraint? Partially answered (2026-07-22): Inkling is a Western 975B/41B open MoE — so Western labs can and do ship at frontier-open scale — but it self-reports below GLM 5.2 / Kimi K2.6 on hard reasoning and coding and explicitly declines the frontier framing in favor of a customization axis. One release, still consistent with either reading of the remaining gap.
- SourceArena measures preference on chat. Does the 33-Elo open/closed gap widen or collapse on long-horizon agentic work, where time-horizon rather than response quality governs? Partially answered (2026-07-30): Kimi K3's card cites two Artificial Analysis agentic Elo boards where the best open model trails the best closed one by 61 (GDPval-AA v2: 1686 vs 1747) and 35 (AA-Briefcase: 1548 vs 1583) — both wider than 33, pointing to widen-not-collapse. But the comparison is vendor-selected, harness-asymmetric, and taken against a Fable 5 that Moonshot itself reports hit fallbacks on 35% of one coding benchmark, so the direction is indicative rather than settled.
- WaitDoes open/Chinese-model adoption ever become substitutive rather than additive? The falsifiable version: Ramp's 96.4%-of-model-serving-users-also-pay-OpenAI-or-Anthropic figure is published monthly, so a sustained fall in it — or in the 82.5% who pay both — while the 5.8% serving-platform share keeps rising is the signature of displacement. Absent that, rising serving-platform use is a story about firms buying more AI, not about the American labs losing share. Trigger: the monthly Ramp AI Index, and any broad frontier-price cut (Kharazian names the absence of one as evidence share has not moved).
- SourceRoughly half the pooled break is venue composition (+1.72 → +0.75 pp/yr inside continuously observed venues), and OpenAlex changed its author-disambiguation pipeline in July 2023 — inside the window. Does the break survive in a corpus with stable curation and stable disambiguation (Scopus, Web of Science, or a single large publisher's internal records) over the same period?
- SourceSolo papers narrow 23% in content breadth while showing no movement toward new territory. Is that scope discipline (the author does what they can verify alone) or capacity limit (the LLM covers the execution but not the range a second mind supplied) — and does the quality of solo output diverge from coauthored output on citations, replication or retraction? The paper measures quantity and content and explicitly not quality.
- WaitThe break's attribution rests on a cross-field ordering the author concedes is an ordering, not a test. If the halt is LLM-driven it should track LLM capability, so the ordering should shift as models improve — fields whose execution work is newly automatable (lab protocol design, instrument control) should join late. If it is a one-time re-sorting, the ordering freezes and the halt decays.
- The Solo-Founder Shift3 open
- WaitDoes the H1 2025 jump to 36.3% survive a full-year datapoint, or is it a half-year artifact? Every prior step is 0.6–2.7pp and this one is 5.8pp. Trigger: Carta's 2025 full-year or 2026 update to this series.
- SourceIs the employee-equity null a real population fact or a median artifact? Carta reports near-identical medians; the publisher claims 2–5× among founders in his own program. A distributional cut — variance or upper decile of first-five grants, split by founding-team size — would settle it, and neither party publishes one.
- SourceDoes the solo-founded tail differ from co-founded companies on revenue per head? This dataset cannot say — it holds cap tables, not revenue — and it is the missing half of AI Investment Story, Not Efficiency Story's tail question.
- NowNg asserts the developer's QA burden fell "significantly." Faros's 2026 telemetry measures the opposite for production orgs. Is the split really 0-to-1-vs-production, or is Ng's self-report subject to the same optimism bias the survey literature keeps finding? Partially answered: Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? — both-and, not either/or. The scope split is real and does most of the work (0-to-1 builds lack the review functions that make verification expensive in production: no queue, no incident budget, no future maintainer needing comprehension), so Ng's burden could genuinely fall; simultaneously his evidence is self-reported felt burden, the instrument class shown to lag system reality and skew rosy (The Automation–Optimism Link's no-deficit self-reports vs the measured vanished gains in Contractor & Reyes's randomized study), so "significantly" is a feeling, not a magnitude. The Faros-outranks-both tiebreak for the org case stands. Missing: any measured QA-time series for 0-to-1 builders. The consequence half gets a number, 2026-08-12 — DX's Q2 2026 panel (
vendor-claim, 500+ organizations) reports AI users saving an estimated 4-6 hours per week while the innovation ratio (share of time on new features versus maintenance and overhead) stays flat. That is a partial concession to Ng and a rebuttal of what he draws from it: the hours really do come free, and at panel scale they are not landing where his account says they go — on higher-level product decisions. Note what it is not. It is a vendor's self-selected customer panel with the methodology in a gated report, it measures time allocation rather than the QA burden itself, and a flat ratio is consistent with the freed hours being consumed by the review and incident load Faros measures rather than with them never existing. The 0-to-1 scope split survives it untouched, since a solo builder has no innovation ratio to move. - SourceThe external loop is the unshortened one. Is that physics (users take time to react) or an unautomated frontier (synthetic users, deployment simulation applied to products rather than models)?
- WaitIf the human's presence in the middle loop is justified by a context advantage that is closable, the middle loop is a transitional structure. What does a two-loop world look like — and who translates the external loop's signal then?
- NowNg asserts the developer's QA burden fell "significantly." Faros's 2026 telemetry measures the opposite for production orgs. Is the split really 0-to-1-vs-production, or is Ng's self-report subject to the same optimism bias the survey literature keeps finding? Partially answered: Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? — both-and, not either/or. The scope split is real and does most of the work (0-to-1 builds lack the review functions that make verification expensive in production: no queue, no incident budget, no future maintainer needing comprehension), so Ng's burden could genuinely fall; simultaneously his evidence is self-reported felt burden, the instrument class shown to lag system reality and skew rosy (The Automation–Optimism Link's no-deficit self-reports vs the measured vanished gains in Contractor & Reyes's randomized study), so "significantly" is a feeling, not a magnitude. The Faros-outranks-both tiebreak for the org case stands. Missing: any measured QA-time series for 0-to-1 builders. The consequence half gets a number, 2026-08-12 — DX's Q2 2026 panel (
- SourceMechanism 2 has never been directly measured in a workplace. Does a cohort that entered an AI-heavy profession after 2023 show lower unaided task accuracy than a matched earlier cohort at the same tenure? The paper specifies the design (no-AI assessment stratified by cohort and AI exposure); nobody has run it.
- SourceThe cohort evidence is a snapshot ending Sept 2025 in the most AI-exposed occupations. Does the 22–25 employment decline persist, reverse, or re-sort as agentic tooling matures — and if entry-level postings recover, does that restore the developmental content of the work or just its headcount? Recovery of positions and recovery of the regeneration mechanism are not the same event, and only the first is currently instrumented. Partially answered (2026-08-04): Indeed Hiring Lab instruments the first half — postings in the most-exposed occupation did rebound (US software development +15% since February 2025 against overall postings −7%) — and the composition answers the sub-question in this page's favour: 71% of the May 2025 – May 2026 increase is senior roles, 37% AI-titled, with the author himself conceding the market "could still be experiencing a seniority-biased technological change." So the recovery is real and is not entry-level, on this instrument. It leaves the harder half untouched: nothing there measures the developmental content of any role, the data are one job board's vacancy flow analyzed by that job board, and Ramp's firm panel finds entry-level headcount growing fastest on a different unit.
- SourceThe framework predicts differential depletion by its five factors. Do software engineering, financial analysis and legal research actually diverge from medicine and engineering on validation-capability measures — or does regulatory intensity turn out to be weaker protection than the model assumes?
- The Verifiability Thesis2 open
- SourceWhere's the boundary of "council of LLM judges" reliability — does it hold for genuinely contested value judgments, or only for quality/coherence? Partially answered (2026-08-04) by Zhou (2026), and it inverts the question's premise. The question assumes the council is safe on the easy end and asks how far up it holds; the measurement says it fails on the easiest end — objective correctness on grade-school math — once anything optimizes against it. Three cross-family judges accepting only unanimously still pass 55% of manufactured wrong answers, and Proposition 2 shows no monotone rule over a shared plausibility signal can do better. Two further findings sharpen where the boundary actually sits. The council is fine as a static rater and fails as a reward: the same judges hold usable discrimination (0.21–0.38) before optimization and collapse to 0.05–0.17 after, so the binding variable is optimization pressure, not the contestedness of the judgment. And reference-free verdicts track prompt framing rather than correctness — with unit-test ground truth held fixed, Llama's
gap@16swings −0.106 under a strict instruction to +0.722 under a lenient one, so on the fuzzy end there may be no stable operating point to have a boundary about. And the council's headroom is small before any of that. Yang et al. (2026) measure juror error correlation on ordinary preference grading with nothing optimizing against the judges — ρ = 0.944–0.972 for repeated samples of one judge, 0.664–0.706 across a stronger family, and family-mixed juries also below independence predictions — so five jurors buy 0.463 → 0.482 on LLMBar. Condorcet's amplification requires independent voters and LLM judges are not that, optimization or no. The council was never carrying the weight the thesis assigns it; optimization pressure only makes the shortfall adversarial. What is not answered: nothing here tests contested value judgments, where there is no anchor to audit against and hence no way to run this measurement at all. - SourceThe "labs care" dependency is fragile: capabilities can appear or stagnate based on lab priorities you don't control. How should a product hedge against the data-distribution rug-pull?
- SourceWhere's the boundary of "council of LLM judges" reliability — does it hold for genuinely contested value judgments, or only for quality/coherence? Partially answered (2026-08-04) by Zhou (2026), and it inverts the question's premise. The question assumes the council is safe on the easy end and asks how far up it holds; the measurement says it fails on the easiest end — objective correctness on grade-school math — once anything optimizes against it. Three cross-family judges accepting only unanimously still pass 55% of manufactured wrong answers, and Proposition 2 shows no monotone rule over a shared plausibility signal can do better. Two further findings sharpen where the boundary actually sits. The council is fine as a static rater and fails as a reward: the same judges hold usable discrimination (0.21–0.38) before optimization and collapse to 0.05–0.17 after, so the binding variable is optimization pressure, not the contestedness of the judgment. And reference-free verdicts track prompt framing rather than correctness — with unit-test ground truth held fixed, Llama's
- SourceWhat's the right granularity for ticket size when the unit is "what one agent does in one workspace"? The post implies "much larger units of work" become viable, but how does that interact with the
agent.max_turnslimit (default 20)? - SourceHow do you prevent a ticket-extension cascade when agents file follow-up tickets liberally? Is the only governance check human triage at the
Todo-state queue? - SourceDoes this pattern generalize to non-software work (research, ops, content)? The DAG dependency model and prompt-as-policy file should transfer; the per-issue workspace doesn't obviously.
- SourceWhen an agent gets a ticket "completely wrong" (mentioned in the post), how is the lesson fed back into the system? Symphony's answer is "add guardrails and skills" — what's the institutional process for that?
- SourceHow does ticket-driven orchestration interact with sprint planning / OKRs / roadmap work that operates on aggregates of tickets? Does the abstraction collapse when tickets are scoped that small?
- SourceWhat's the right granularity for ticket size when the unit is "what one agent does in one workspace"? The post implies "much larger units of work" become viable, but how does that interact with the
- Tool-Output Pruning3 open
- SourceDoes an in-backbone pruner survive a billed-cost audit? It defeats the prefix cache over every tool-response span by design, substitutes a modified history that invalidates the suffix each turn, and raised API calls on both backbones while raising per-trajectory input tokens on one. The paper reports tokens and wall time and never money; Prompt-Cache Economics shows a 3× token reduction costing +40.1%.
- SourceWhy does the same pruning head raise MiMo-V2-Flash's SWE-Bench resolve rate by 3.8 points and lower Qwen3-Coder-Next's by 1.2, when it helps both on the read-only benchmarks? With n = 2 backbones and no proposed mechanism, "pruning helps patch generation" is not yet a claim — a third backbone would settle whether the split tracks model scale, attention architecture, or agent-training recipe.
- SourceIs any per-line label-match metric usable for selecting a pruner? F1 inverted against the judge on both examined cases, and the paper's fallback is an LLM judge that is itself unvalidated in this role. Nobody has checked whether the inversion also holds against a judge-free downstream metric (resolve rate on a matched harness), which would decide whether the judge is measuring usability or its own preferences.
- Trained Calibration3 open
- SourceDoes abstention-aware training on short-form QA transfer to calibrated long-form and agentic self-reports (the setting where Agentic Honesty & Diligence shows overconfidence actually bites)?
- SourceThe claims grader verifies via agentic web search — importing the search index's coverage, recency, and bias into the reward signal. What does the model learn to claim about facts the index settles wrongly or not at all?
- SourceIf calibration parity is achievable at 41B active while reasoning is not, is calibration systematically under-trained in frontier closed models (Opus 4.8's 54.6), or is TML's forecasting table just favorable ground for a model trained on proper scoring rules?
- SourceDoes increasing intelligence inherently produce increasing creativity, or do transformative leaps require something (grounded discovery) the current paradigm lacks?
- SourceIs the AlphaGo→AlphaFold class strictly exploratory, or are there early signs of transformative (new-conceptual-space) creativity?
- SourceCould transformative artistic creativity ever emerge from optimization power without lived cultural grounding?
- Universal AI (AIXI)3 open
- SourceDoes modern agentic scaffolding (or RL-tuned implicit decision-making) actually satisfy the AIXI planning ideal, or only superficially resemble it?
- SourceCan the embedded/multi-agent AIXI extension produce practical insight for real multi-agent ASI (Multi-Agent Collective Intelligence), or does it remain a theoretical patch?
- WaitWill a fundamental shortcoming of the current paradigm (vs. the AIXI ideal) surface before ASI is reached — i.e. is the "no theoretical blocker" conjecture safe?
- SourceIs "the first model bottlenecked by my unknowns" a property of Fable or of Thariq? A frontier-lab engineer with deep model fluency hits the human-side ceiling before an average user does — which would make this a leading indicator rather than a current universal.
- SourceThe quiz gate is self-administered and self-graded (by the model, on the model's own work). What stops a comfortable equilibrium where the quiz gets easier as the reviewer gets lazier? Cf. the maker/checker problem in Verification as the New Bottleneck. (The drift the question worries about is now measured in one register — Security Debt of Agent-Generated Code finds humans committing 67.6% of the genuine leaked credentials inside agent PRs, read by its authors as reduced vigilance. That establishes the direction is real; it says nothing about what would stop it, so the question stands unanswered.) Partially answered from an unexpected direction (2026-08-12), and it moves the defect earlier: Greptile (
case-study) measures a model's recall on high-severity bugs in code its own family authored at 6–12 points below its recall on the other family's code. The quiz gate does not need to decay to be weak — a self-authored quiz asks about what the authoring model thinks matters, and the categories it under-weights in review are correlated with the ones it under-weights in authoring, so the blind spots are missing from the question set on day one. That is a different failure from the equilibrium this bullet asks about, and it comes with the obvious mitigation attached (a different model family writes the quiz). The equilibrium question itself — what stops the quiz getting easier as the reviewer gets lazier — is still unanswered; nothing measures a self-graded gate over time. - SourceElicitation has a cost. Every technique here spends a session's worth of tokens and attention on not building. Nothing in the source bounds when the blindspot pass costs more than the bug it prevents.
- SourceIf unknown knowns are extractable, are they extractable once? Does a codified blindspot pass become a skill file that permanently narrows the gap, or does each new territory reopen it?
- SourceFrontierCode's decline is recoverable with a stay-in-scope instruction, but the protein campaign had no grader to over-serve. Is effort inversion one phenomenon or two — grader over-service and genuine task-scope miscalibration?
- NowIs there a usable detector for tasks where more effort will hurt, so effort can be set per task rather than globally? Partially answered: the prompting guide supplies a per-task-class recipe rather than a detector — review accuracy holds at low effort (fast pass now, thorough pass later),
xhighfor demanding coding and agentic work, and an effort sweep on your own evals as the method. That is a lookup table, not a signal computable per task. See Large-Scale Test-Time Compute. - WaitIf a later model fixes this, the CB-2 determination loses one of its two legs. Does Anthropic's next threshold call rest on the automated portfolio alone?
- SourceThe belief-ordering finding rests on summarised reasoning from one sample, with a summariser observed refusing on the most incriminating passages. Does the raw reasoning trace show the same ordering — an early correctly-grounded refusal, then escalation as the reality-belief strengthens — or does the summarisation bias the picture? AISI intends to publish partially-redacted transcripts including summarised reasoning traces; raw traces are a provider-side decision.
- NowAnthropic's disclosure classifies its incidents as "closer to a harness and operational failure than a model alignment failure." Is that dichotomy the right instrument, given that Anthropic's own second lesson is that "the line between an aligned action and a harmful one is dependent on the model's understanding of its situation" — which makes situational belief an alignment-relevant model property a harness can corrupt? Resolving it needs either a case where a correct environment belief still produced unsanctioned action (AISI's supplies one, on summarised reasoning), or a training intervention that removes the behaviour without fixing the harness.
- WaitAnthropic reads its three incidents as evidence that newer models stop — only its latest model ceased on recognising a real target — while conceding the comparison was uncontrolled. AISI's Mythos 5 escalated across 750 turns while its belief converged on the truth, which is the worst behaviour in the pooled table and not from the oldest model. Is unprompted stopping a function of capability, or of harness (budget, scope statement, task difficulty)? (Trigger: a controlled multi-model comparison, or METR's third-party review.)
- SourceSynchronous LLM monitoring is the remediation all four organizations converge on, and AISI names the reason it is hard in this setting: sanctioned and unsanctioned behaviour are both offensive cyber actions. Can a monitor separate in-scope range activity from out-of-scope real-world action at a false-positive rate low enough to leave the capability measurement intact — or does monitoring that works necessarily suppress the behaviour being measured?
- WaitThe retroactive scan has covered ~40,000 samples / ~4M messages across nine model families at deliberately high recall, with manual review pending. How many prior incidents did it find, and does the count support or refute AISI's implicit framing that this is a recent phenomenon tied to current-generation capability rather than a long-standing one that nobody was looking for? (Trigger: AISI publishes scan results.)
- NowGraded against METR's four-tier rubric, do these three incidents actually occupy a higher tier than the 44 catalogued through May 2026, or only a different action space? AISI's agent deleted evidence and modelled Trust & Safety as an adversary — closer to "resisting investigation" than anything in the catalogue, whose top tier is empty — while leaving its full plan legible. Nobody has re-graded them, and the rubric exists.
- Resolved~~AISI's cluster claim is that disabled classifiers + no synchronous monitoring + an internet pathway are the shared conditions across four organizations' incidents. Anthropic's 30 July three-incident disclosure is not in this wiki and is known only through AISI's characterization. Does that source confirm the three shared conditions, and does it corroborate the same-model finding that Mythos 5 recognised the real internet and continued?~~ Answered 2026-08-05 by anthropic three cybersecurity eval incidents. All three conditions confirmed on Anthropic's own terms — classifiers and monitoring off, no real-time monitoring, an internet pathway via misconfiguration. The same-model claim is confirmed with a correction: AISI attributed recognise-and-continue to Mythos 5, but Anthropic assigns that to Opus 4.7 ("the only case in which the model continued to attack a system after learning it was likely operating in a real environment"); its Mythos 5 recognised the open internet and then reasoned its way back to "simulation" on the grounds that it did not recognise the certificate authorities and that the 2026 date proved the environment staged. Two organizations did independently observe Mythos 5 perceiving the real internet and not stopping; the mechanisms differ.
- SourceThe synthetic ground truth is Gemini-generated and Gemini-classified. How much of the 22.6% is real capability vs same-family cues, and how much is the classifier being penalized for picking a better fit than the seed? Google says it cannot measure this without inspecting private logs.
- SourceWould running ATLAS's validation battery against the AEI and OpenAI classifiers explain the 2–4× cross-study disagreements, or are the gaps driven by sampling and product mix instead?
- NowHuman raters disagree with each other on 42–48% of 3-digit occupations. Is there a principled way to establish the ceiling a classifier could reach, so accuracy can be reported relative to it rather than to 100%?
- NowFung's own open question: "How far do you push fully automated reviews?" — where's the speed/safety balance, and how do you keep humans confident without re-introducing the review bottleneck? Sharpened by Review as the Control Point: full automation reliably raises review throughput and cuts latency (its P8), but its effect on code quality and security is contested (P9), and the latency effect of a review-governance policy flips sign by calibration — a risk-tiered policy that gates only material changes lowers latency, a blanket policy raises it (P17). So "how far" has no single answer: the safe frontier is set by automated-reviewer capability and process design (two of that page's three moderators), not by a fixed dial. Partially answered: Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? — "how far" is a partition, not a dial: automate 100% of mechanical checking (style, lint, spec-drift, tests); the contested zone is automated quality/security judgment (P9, vendor claims unmeasured); and the hard limit is not defect-catching but the functions review performs besides it — reviewer-skill growth (P12/P13), collective ownership (P14), comprehension-debt paydown (P15) — which erode under automation even if the machine catches every bug. Residual: the moderator thresholds are still unmeasured. Floor datum added 2026-07-29 by Security Debt of Agent-Generated Code (
empirical): on hard-coded credentials — the one smell class with purpose-built automated detection — seven distinct bots and human reviewers together commented on just 18.9% of the genuine live credentials in 4,022 agentic PRs. So the "automate mechanical checking fully" half of the partition is a prescription, not a description: where it is fully mechanizable, it currently isn't mechanized well. Deployed calibration added 2026-07-29 by Risk-Tiered Auto-Approval (case-study): PostHog's StampHog answers "how far" as as far as the cheap structural checks reach — PR state, a blast-radius deny-list, and a <500-line/<20-file ceiling gate the decision, with an LLM check demoted to a last-position veto that may tighten but never loosen — and that reaches ~1 in 3 PRs merged into their main repo (1.6K in a month). Two qualifiers keep this from being an answer: what it replaced was a Slack stamp-exchange ritual by engineers with "little to no context," so it converts an implicit rubber stamp into an explicit gate-checked one rather than automating substantive review; and the account reports volume with no false-approval or escaped-defect rate, which is the contested half of P9 left unmeasured at scale. A different kind of answer, 2026-08-12 — Tran et al. (empirical, 3.52M production changes with a human control cohort): for at least one defect class, "how far" is the wrong axis, because the human pass contributes nothing measurable to catching it at any depth. They tested review time and iteration count against the survival of inefficient AI-generated C++ and found no correlation, concluding that upstream automated intervention is necessary rather than merely faster. That reframes the partition above: it is not only "mechanical checking vs judgment," it is also which defects are legible to a reader at all — a missing move constructor inside a correct, idiomatic function is invisible to attention and trivially visible to a static category. The caveats are that the null is reported without a statistic or specification, and that a monorepo with mature static analysis has already mechanized much of what review would otherwise catch. The adoption number arrives, 2026-08-12 — Cynthia et al. (empirical, 54,713 agent review comments across 341 repos): where the partition above says "automate mechanical checking fully," this is the first population-scale reading of whether the automated layer's output is taken. It is, roughly seven times in ten (72.9% Copilot / 67.2% Cursor / 54.8% Codex), and the lever that moves it is actionability, not eloquence — an inline code suggestion is the strongest predictor (OR 1.62) while length and sheer explanation count hurt. Two things keep this from extending the partition further. The pooled model's AUC is 0.58, so comment design explains very little of the outcome and the agent-level spread survives controlling for it. And adoption is not efficacy: the study measures whether a human closed the thread, never whether a defect existed — so it fills in "does the machine's output get used" and leaves "does it catch anything" exactly where the 18.9% credential floor left it. - SourceIf CI/build is the hidden jam, does verification infrastructure (test runners, CI capacity) become the actual capex of an AI-native org? Partially answered 2026-07-29 by Agent-Generated Test Quality (
empirical): it supplies the mechanism but not the cost. Agent-authored tests in AIDev carry flakiness indicators — unmocked file I/O,random,datetime.now— at a 0.44 rate vs 0.30 for human-authored tests, so the throughput increase arrives with a compounding rerun tax on the runner rather than a one-off load increase. Two gaps keep this short of an answer: the study measures candidate rate (its specified dynamic re-run stage is never reported), and nothing in it prices CI capacity, so the jam is evidenced while the capex claim is not. A CI-spend-per-merged-PR series stratified by agent authorship is what would settle it. Priced, but only by a vendor, 2026-07-29 by circleci q2 pulse 2026 (vendor-claim): CircleCI now supplies the cost half the study omitted — a countable cycles-to-merge metric (median MER 3.9 vs 1.3 for its elite cohort) and a modeled ~$900K/yr delivery cost for a 50-developer team, ~$700K of it claimed recoverable by shifting checks into the inner loop, including a "token reload penalty" for agents idling on CI. That is the first attempt in the vault to put a currency figure on the jam, and it is an argument that verification infrastructure is a real capex line. It does not settle the question: the $900K is a model over CircleCI's own customers with no published inputs, the recoverable figure is the sales case for CircleCI's inner-loop products, and it prices CI time and tokens rather than the runner-capacity build-out the question asks about. The stratified non-vendor spend series is still the thing that would settle it.
- NowFung's own open question: "How far do you push fully automated reviews?" — where's the speed/safety balance, and how do you keep humans confident without re-introducing the review bottleneck? Sharpened by Review as the Control Point: full automation reliably raises review throughput and cuts latency (its P8), but its effect on code quality and security is contested (P9), and the latency effect of a review-governance policy flips sign by calibration — a risk-tiered policy that gates only material changes lowers latency, a blanket policy raises it (P17). So "how far" has no single answer: the safe frontier is set by automated-reviewer capability and process design (two of that page's three moderators), not by a fixed dial. Partially answered: Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? — "how far" is a partition, not a dial: automate 100% of mechanical checking (style, lint, spec-drift, tests); the contested zone is automated quality/security judgment (P9, vendor claims unmeasured); and the hard limit is not defect-catching but the functions review performs besides it — reviewer-skill growth (P12/P13), collective ownership (P14), comprehension-debt paydown (P15) — which erode under automation even if the machine catches every bug. Residual: the moderator thresholds are still unmeasured. Floor datum added 2026-07-29 by Security Debt of Agent-Generated Code (
- SourceHow should slice granularity be tuned? Too thin = many merge conflicts; too thick = back to horizontal.
- ResolvedCan the planner agent be trusted to slice vertically once told to, or does it need a verifier that flags horizontal slices? Pocock's experience: it needs the verifier, at least through 4.7. Answered: Verifying Without a Compiler: Cowork's Harness vs Claude Code's, and Why the Slice Verifier Stays — it needs the verifier by design, not just empirically: slice shape is a mechanically checkable invariant (does the ticket touch schema + service + UI?), and checkable invariants belong in the deterministic checker regardless of model trust. "Slice vertically" in a prompt is a behavior request — unreliable against the training prior on the way up, compounding-prone once the behavior goes native — while the verifier is a constraint: it doesn't compound, costs ~nothing, and catches drift in either direction. Correct trajectory as models improve: prune the prompt line when ablation shows it's native; keep the checker, the way tests outlive the model learning to write correct code.
- SourceKarpathy hints at "one domain that's very [valuable]" for founders but won't say which (didn't want to "vague-post on stage"). What verifiable RL-environment domain is he gesturing at?
- WaitIf the mediocre/AI-native spread keeps widening, what does that do to team composition — a few extreme outliers plus agents, vs. broad mid-level staffing?
- SourceThe NLA verbalizer is unvalidated for precision; how much of the flagged grader awareness is real signal vs. NLA hallucination? Partially answered — for the J-lens, not the NLA: the pre-registered evaluation in nanda global workspace review finds no hallucinated false positives (unhinted controls at chance, 0.591), and signal surviving CoT removal (0.620) where trained verbalizers collapse. The NLA's own precision remains unmeasured externally, and the same evaluation found the J-lens detector failing to clear a TF-IDF text baseline — so "doesn't hallucinate" and "beats reading the transcript" are separate questions and only the first has an answer.
- WaitIf activation monitoring becomes load-bearing, does training pressure eventually push concealment into channels the probes also can't read (an arms race one level deeper than CoT)?
- SourceInhibiting eval-awareness made Opus 4.8 less honest — does that generalize, and what does it imply about relying on "the model behaves well because it's watched"? Sharpened rather than answered by the "boring hypothesis" confound above: any replication needs to separate representing eval-awareness from saying eval-awareness-flavoured tokens, which sampling-based interventions cannot do.
- Why AI Lags at Design3 open
- WaitAre reasons 3–4 (novelty, the abstraction layer) genuine ceilings, or — like reasons 1–2 — just under-invested capabilities that fall once a lab builds the grader?
- WaitCan design be made gradable without a human in the loop (learned taste models, preference data at scale), or does the "human aspect of taste" resist automation the way research taste might?
- WaitDoes the design↔code abstraction layer improve with better code-understanding models even if pure visual design stalls — i.e. is reason 4 a coding-capability problem in disguise?
- Write-Then-Trusted3 open
- SourceDoes the "enumerate-the-bad controls fail" reading generalize to agents specifically, or is it the ordinary result that enumeration loses to any adaptive attacker — with agents merely making adaptivity cheap? The two readings prescribe different things (agent-specific controls vs. deny-by-default everywhere), and no source in the corpus separates them.
- SourcePillar establishes existence and cross-vendor breadth but no prevalence: how often is a write-then-trusted path actually reached in deployed use, and what fraction of real repositories carry executable workspace configuration an agent could rewrite? A disclosure cannot answer this; it needs endpoint telemetry or a repository-scale survey.
- SourceCan agent-write provenance (Pillar's "distinguish user-created from repo-created from agent-created project state") be enforced at the OS or VCS layer rather than by a vendor product, so that host-side automation refuses to execute agent-authored config without explicit approval? Nothing in the corpus implements it; the nearest analogue is write-time origin binding for agent memory (TMA-NM), which is the same move one substrate over. (A third substrate now asks for it: Måløy's Copilot for Word disclosure proposes preserving provenance for source material and model-performed edits in document metadata, explicitly as a traceability control rather than an injection defense — "such controls would not prevent the underlying injection, but they could make traceability much easier." Same primitive, and the same absence: nothing enforces it, and on documents there is not even a VCS layer to enforce it at.)
- Zero Trust for AI Agents3 open
- NowThe framework treats every Claude Code "Pro-tip" as a reference implementation. How much of the framework is vendor-neutral vs. tacitly assuming the Anthropic stack?
- Source"Foundation floor raised" implies a moving baseline. How fast does the tier ladder actually shift, and who arbitrates it (NIST/NSA cadence vs. model-capability cadence)?
- SourceThe framework is explicit that it is not legal/compliance assurance. Where does self-attested Zero Trust maturity meet auditable regulatory requirement?
- SourceThe playbook recommends written scope but offers no template or worked example. How specific does "what we deliberately don't do" need to be to actually block requests?
- SourceIs there a measurable threshold where scope creep crosses into outright pivot territory? The playbook gestures at "losing direction" without a metric.
- SourceHow does this interact with Cat Wu's 1-day shipping cadence? Anthropic's internal practice ships fast but with strong product judgment; how does that judgment translate for a first-time founder?