Agent Systems, in order.

Open questions 138 open

• Agent Context Files5 open
  • WaitWill the role split converge on Hermes's explicit project/personality separation, or stay folded into a single file as in Claude Code? A separate SOUL.md-style personality layer seems strictly better for multi-project users but adds a file to maintain.
  • WaitIs there a natural ceiling on the layering (project → workflow → spec → constitution), or does each new autonomy surface spawn another context-file tier?
  • SourceDoes the universal system-prompt slot cost anything? Every vendor on this page injects context files into the system prompt, and the only controlled measurement of that choice (prompt design at scale) finds placement is a larger lever than format with a model-specific sign — helping two models, hurting two. Falsifiable cheaply: render the same CLAUDE.md / AGENTS.md into the first user turn instead and measure adherence per model. (Genkit's skills middleware is a fourth vendor making the same choice — frontmatter metadata injected into the system prompt at init — which widens the premise without touching the question.)
  • SourceDoes context the agent provably cannot infer move correctness, where generic convention context does not? Khatri's null is scoped to naturalistic style-guide content on repositories the agent can read in full, and his failure triage says the gating deficit is implementation skill. The discriminating experiment is his own stated gap: rerun the ablation with purpose-built, task-specific context encoding a fact absent from the codebase (an undocumented external API contract, a deployment invariant, a "this test is flaky for reason X" note) and see whether near-misses flip. If they don't, the practitioner implication hardens from "generic files don't pay" to "context files don't pay for correctness at all."
  • SourceMetadata-injection discovery relocates the instruction-count ceiling onto the skill catalog rather than removing it: every installed skill's description is resident from initialization, so a large enough skills/ directory should floor adherence before any skill body loads. How many resident descriptions does that take, and does use_skill selection degrade before or after all-rules compliance does? Falsifiable with prompt design at scale's harness pointed at N skill frontmatters instead of N rules.
  • ResolvedHow should context files and bounded memory files interact when they disagree? Memory is lossy and cache-delayed; the context file is authoritative but static. Which wins, and when? Answered: When Knowledge Layers Disagree: Context Files vs Memory, and Conflicting Sources at Compile Time — split by disagreement type. Policy: the context file always wins — it is the human-reviewed, git-versioned high-integrity channel, and agent-written memory's recency cannot confer authority (a memory item contradicting policy is indistinguishable from staleness or poisoning, per the TMA-NM laundering theorem). Facts: neither wins — both are caches over reality; verify against the repo/live state (the code-as-source-of-truth arbiter) and repair the stale cache. Always: log the conflict for the lint/pruning pass (the deviations-log pattern) instead of silently breaking the tie, and let writes flow only down the integrity ordering — memory never modifies the context file; the context file legitimately bounds memory.
• Agent Harness Engineering3 open
  • WaitHow does architectural coherence evolve over years in a fully agent-generated system?
  • SourceAt what codebase scale does the AGENTS.md-as-table-of-contents approach need to be replaced with more sophisticated context routing?
  • SourceHow generalizable are these web-app-focused findings to other domains (scientific research, financial modeling)?
  • ResolvedDoes a single general-purpose coding agent outperform a multi-agent architecture with specialized testing, QA, and cleanup agents? Answered: Single General Agent vs. Multi-Agent Coding Architecture — no single winner as posed; a single general agent overtakes a bespoke hand-engineered multi-agent system as models improve (The Bitter Lesson), but a monolithic-context agent loses to role separation (fresh-context explorer/reviewer + independent grader), which persists because it fixes structural constraints (quadratic attention, Goodhart), not model weakness.
• Agent Loop Pattern3 open
  • WaitWhen the model schedules its own loops (4.7 behavior), who owns the budget? Boris answered "the model just decides" — but that pushes cost discipline into the model's training, not the harness.
  • WaitDoes a loop with a smart enough model still need a Kanban backlog, or does the model choose its own next task from raw goals?
  • NoteLoop output review is now Matt Pocock's confessed bottleneck — "we just need to be ready to be doing more code review."
• Agent Quality Flywheel3 open
  • SourceBoth demo cycles fixed agents with instruction-level bugs and showed large one-cycle gains. What does the loop look like on failures that need tool, memory, or architecture changes — does "several iterations before metrics move" dominate in practice?
  • SourceThe custom rubric is authored by the same coding agent that will later propose fixes. Metric choice is upstream of grading — does decoupling need to extend to who defines the metric, not just who scores it?
  • SourceSynthetic User Simulator scenarios bootstrapped the whole first cycle. How much of the 21%→5% delta survives on real-traffic distributions (the representativeness gap Production-Sourced Evaluation names)?
• Agent-Authored Harness Optimization4 open
  • SourceDoes agent-authored harness evolution actually beat simple test-time scaling, and does it generalize to held-out tasks? Partially answered — and the two halves now point opposite ways. Test-time scaling: answered, negatively. rethinking harness evolution evaluation (empirical) supplies the full-suite budget-matched arm this bullet named as missing — four methods at K = 5 on Terminal-Bench 2.1 across three frontier models, with and without unit-test feedback — and harness evolution finishes below plain parallel sampling everywhere and below do-nothing on average without unit tests (67.4 vs 68.2), with its pass@5 gain traceable to multi-attempt selection rather than pass@1 capability. Held-out generalization: contested, not settled. harnessbank harness self evolution credits six of seven sealed per-domain tests at z ≥ 1.96 (+9.2 to +15.4pp); Wang et al.'s 45/10/34 split on Terminal-Bench 2.1 returns +0.6pp average and +0.0 on GPT-5.4. The proposed discriminator is baseline headroom (HarnessBank's TB2 arm evolves a 36.1-point 27B backbone; these are frontier models at 63–75), which is testable and untested. Still missing entirely: transfer to a different benchmark, which no source measures.
  • SourceIs the negative result a property of harness evolution or of Terminal-Bench? Wang et al.'s §5.2 names two conditions a fair test needs — substantial headroom above current agent scores, and a benchmark whose performance genuinely depends on the harness (specialized tools, skills, workflows) rather than one where "a shell tool and a basic prompt already suffices." Terminal-Bench satisfies neither for frontier models, and it is the benchmark essentially the whole harness-evolution literature reports on, including HarnessBank and Cline. Falsifiable directly: run the same four budget-matched arms on a harness-sensitive suite and see whether the ordering inverts.
  • SourceHow much of the 77.5%→88.8% survives on an agentic benchmark Cline has not been hill-climbing for six months? Every fix is described as a general harness improvement, which is a transfer claim, and no transfer measurement is reported. (HarnessBank does not touch this: it measures within-domain held-out tasks, not cross-suite transfer.)
  • SourceDoes the semantic quality-diversity archive earn its complexity against a plain greedy keep-the-best loop, holding the significance gate fixed? HarnessBank ablates the gate but never ablates the gene bank; its archive evidence is observational (credited harnesses stack mechanisms from more than one cell), not a controlled arm.
  • ResolvedIs there a stated test that separates narrow scaffold hill-climbing from the recursive self-improvement the term names — e.g. capability transfer to unrelated tasks, or improvement that compounds without a human re-issuing the brief? Answered 2026-08-04: What Makes a Self-Improvement Artifact Transfer? states the test: name what the improvement artifact is fitted to, then measure whether gains survive outside that scope. HarnessBank's two separators (cross-model transplant near-zero off the matched pathology and -15.7 wrong-way; termination at the 10-round floor vs phantom progress in 62–76% of ungated post-convergence rounds), Wang et al.'s compute-side separator (at matched budget the loop does not beat sampling five times), and Caltech's orthogonality result (a domain-fitted artifact compounds portably while the model improves not at all) instantiate it: solver-fitted gains are maintenance, domain-fitted gains are assets, and RSI would require the gains to live in the solver itself. The stated test's un-run instances remain evidence gaps: cross-suite transfer stays in this page's #oq/source items, the two-artifact controlled test on Knowledge-Centric Self-Improvement, and no source yet runs a loop that re-issues its own brief.
• Agent-Native Infrastructure2 open
  • WaitWho builds the agent-native rewrite of the long tail of human-facing services — the service owners, or a translation layer (MCP servers, computer-use agents) on top?
  • Agent-to-agent negotiation needs trust, identity, and accountability primitives that don't exist yet. What's the protocol layer, and who governs it? Partially answered: AIMS (IETF draft-klrc-aiagent-auth-03) proposes the protocol layer — agent-to-agent is just workload-to-workload, so WIMSE/SPIFFE identifiers, OAuth 2.0 delegation + token-exchange chaining, and OpenID Shared-Signals eventing (drawn from IETF/CNCF/OpenID) supply identity, delegated authority, and auditable accountability; a tool endpoint "may itself be implemented by another AI agent." But who governs it stays open: AIMS is an individual submission with no IETF WG consensus, profiling a stack of specs that are themselves still Internet-Drafts — the primitives are proposed, not ratified or arbitrated. Sharpened (2026-07): the who-governs-it answer is concretely plural — the identity/authentication/delegation slice is IETF-track (AIMS), while the authorization slice is OpenID-Foundation-track: its AuthZEN Working Group approved the AARP (a prerequisite/approval pattern generalizing CIBA — "not yet, here is what is required") and COAZ (MCP-tool-authorization) profiles as Working Group Drafts on 2026-06-15. So the protocol layer is being standardized across multiple bodies (IETF for who-you-are + how-authority-delegates; OpenID for whether-a-call-is-allowed + what-must-precede-it) — moving, but with no single arbiter and no ratified cross-body composition yet. (Standards-announcement, practitioner-opinion — proposed drafts, not settled specs.)
• Automated Failure Attribution3 open
  • SourceEvery trace is one injected error into a run that otherwise succeeded, so a unique decisive step is guaranteed to exist. Does attribution accuracy survive organic failures with multiple interacting causes and no single flip-the-outcome step — where the honest answer is often "three things were marginal and the fourth tipped it"? The human panel found only 2.0% of traces had no clear decisive error, but that is a property of the construction, not of agent failures. Until this is measured the 73.9 / 25.3 numbers are an upper bound of unknown tightness.
  • SourceHumans and models are never scored on the same task. Annotators ratify or correct a supplied label (94.0 / 90.0 / 90.0, κ = 0.73); models predict from scratch (73.9 / 57.5 / 22.2). So the ~20pp step gap is not a measured human ceiling, and could be much smaller or much larger. What is human from-scratch attribution accuracy on these traces? Cheaply falsifiable with the existing corpus and a blind annotation arm.
  • SourceThe base agents were chosen for unencrypted, interpretable reasoning chains because the judge must read intermediate reasoning to localize the decisive step. Frontier deployments increasingly summarize, encrypt, or discard those chains. How much attribution accuracy survives when the decisive step's reasoning is not in the trace — and is the loss concentrated in the step metric or the error-mode metric? Answerable today by re-running the benchmark with reasoning content stripped or replaced by provider summaries.
• Build for the Next Model2 open
  • SourceHow do you tell a "wait for the model" gap from a durable-harness gap before the next release? Get it wrong and you either ship vaporware or build a crutch you'll delete. Partially answered: What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful? — the survivor taxonomy is the classifier: gaps in behavior/capability (task-prior structure) are "wait for the model" gaps; gaps in boundaries, org-specific record, identity, deployment structure, or human-facing legibility are durable-harness gaps that no release will close. The pre-release tell is what the fix would encode.
  • SourceDoes the strategy generalize outside frontier labs, who have privileged visibility into the next model? An external team is betting on a release it can't see.
  • ResolvedThe bet depends on a reliable release cadence and a forecastable capability curve (Task Time-Horizon Scaling). What happens to "build for the next model" if model improvement stalls (the stalled-but-diffused future)? Answered: What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful? — it degrades gracefully, because correctly practiced it is a cheap call option on the release cadence: a stall costs the premium (the prototype portfolio expires unexercised), not the firm, provided market validation was kept separate from the capability bet. Three cushions: Latent Capability Overhang keeps effective capability rising post-stall (mine the model replaces wait for the model); the previously-wrong move — engineering the compensating crutch — reclassifies as correct ("too AGI-pilled" correction becomes the standing posture); and competition shifts to the durable layers that never migrated inward. The bet only fails hard under miscalibration: shipping products whose core loop depends on absent capability, which is vaporware whether or not the cadence holds.
• Claude Code Auto Mode4 open
  • SourceWhat false-positive rate does the classifier have on routine-but-aggressive refactors (e.g., large-file renames, rm of build artifacts)?
  • SourceHow well does the classifier generalize to custom tools / MCP servers where it lacks environment context?
  • SourceIs the classifier's decision boundary documented/stable enough for security-sensitive orgs to certify, or is it effectively a black box whose behavior drifts with updates? Partially answered (2026-08-04): the changelog supplies the two facts a certification argument would start from — the classifier defaults to Sonnet 5 for external sessions, and it is "validated on the session's first request and pinned for the session" (v2.1.210), so the boundary cannot move under a running agent. That closes intra-session drift and closes nothing else: no decision boundary is documented, the scope of what the classifier adjudicates widened in v2.1.218 (taking over the dangerous-rm, background-&, suspicious-Windows-path and unprovable-read-only-Bash checks from static analysis), and release-to-release drift is unaddressed and now covers more surface than before.
  • SourceDoes extending auto mode to API users change its calibration — is the classifier retrained for automation-heavy use, or held constant?
  • ResolvedCompared to OS-level sandboxing (mentioned in Claude Code Best Practices alongside auto mode), what's the defense-in-depth story? When should both be layered? Answered: Classifier Gates vs OS Sandboxing: The Defense-in-Depth Story for Auto Mode and Cowork — they are different control kinds with disjoint blind spots: the classifier is a semantic gate (judges intent the sandbox can't see — within-capability harm over allowed channels) and the sandbox is a structural barrier (contains the classifier's two documented failure modes). They fail independently by mechanism, making the stack real defense-in-depth rather than correlated friction. Layer both whenever the agent holds reach beyond the sandbox boundary (live creds, MCP to real SaaS), runs unattended, or reads untrusted input; sandbox-only is legitimate when containment is total (the Hermes Agent container-is-the-boundary design point); classifier-only is a stopgap for interactive low-stakes local work — never for unattended runs.
• Claude Code Best Practices3 open
  • SourceDoes the instruction-count ceiling hold for conditional policy, where only a handful of rules bear on any given turn? Every rule in Eliav's experiment applies to one generation simultaneously; a CLAUDE.md is mostly situational ("when editing migrations, …"), so N=80 is a floor on the harshest possible loading and says nothing about a 200-rule file of which five fire per turn. Falsifiable directly: hold the applicable subset fixed and grow the inapplicable remainder.
  • SourceHow does the Writer/Reviewer pattern compare to agent-to-agent review (as in OpenAI's Codex workflow)?
  • SourceWhen does subagent overhead exceed the benefit of context isolation? Partially answered 2026-08-03 by codex 0 to 10m users chatgpt work (practitioner-opinion, no measurement) — a task-shape criterion rather than a crossover point. Akshay Nathan (OpenAI): multi-agent modes "are best for when you have tasks that are either incredibly complicated, like open explorations, or very paralyzable… but for most tasks, they don't fall into either of those buckets," so the default should be a single agent. Note the overheads he actually names are neither context nor tokens: rate-limit consumption (Ultra "can use more of your limits," which is why OpenAI moved it behind advanced settings post-launch) and human legibility (sub-agent transcripts hidden by default to avoid overwhelming users — see Shared Harness, Differentiated Surfaces). A practitioner counter-practice in the same episode pulls the other way: Vibhu reports telling every long-running task to "use sub-agents where possible" for wall-clock and for cost, fanning out to cheaper models — which Cost-per-Task Over Cost-per-Token argues is the wrong default. Second instance (2026-08-04): Willison runs the same practice with the tier choice itself delegated — "for all coding tasks use your judgement to decide an appropriate lower power model and run that in a subagent" — and reports only that his Fable allowance shrinks more slowly. Two practitioners now default to the fan-out; neither measures it. What remains open is the measured crossover, which no source in the corpus supplies.
  • ResolvedWhat's the optimal CLAUDE.md length before instructions start getting lost? Is there a measurable threshold? Answered 2026-08-04 by prompt design at scale (Eliav, arXiv 2607.19257, empirical) — full treatment at Instruction Compounding. Yes, there is a threshold, and it is count-shaped rather than length-shaped: across five models (including Claude Sonnet 5 and Haiku 4.5), the rate at which every instruction in the prompt is obeyed falls steeply by N≈40 simultaneous verifiable rules and hits zero by N≈80, holding through N=160 and identical across markdown, plain text, prose, and table renderings and across system-prompt vs. user-turn placement. The paper's own prescription is the answer in usable form: 40 simultaneous instructions is a redesign point, not a tuning point — past it, splitting across turns, tools, or a validation pass is the only thing that works, and reformatting is not. This also settles the residual the 2026-08-03 retag left behind (does aggregate size have an independent effect once each line passes ablation?): yes — every rule tested was distinct, non-redundant, and individually satisfiable, so per-line ablation non-inferiority would have cleared all of them and still missed the collapse. Two scope limits carried on the answer: "perfect response" is a strict conjunction, so some of the floor is the arithmetic of ANDing N checks rather than the model dropping the block, and every rule tested is a hard output constraint applied to a single generation. The conditional-policy case a real CLAUDE.md actually presents is now the successor question in Open Questions above.
• Client-Side Agent Optimization5 open
  • SourceHow does combination-level optimization interact with continual model releases? If Claude Opus 4.7 ships next month, does the full Pareto frontier need re-running, or do warm-started bandits adapt cheaply? Partially answered (2026-08-04, by synthesis): What Makes a Self-Improvement Artifact Transfer? — a combination is a solver-fitted artifact (fitted to the current menu's capabilities and prices), so the frame predicts full re-runs rather than cheap adaptation, with the durable residue being assignment rules rather than assignments; the HotpotQA→Cursor inversion below already shows the assignment not transferring while the reconciling rule does. The warm-started-bandit half is an empirical question no source measures.
  • SourceAt what pipeline depth does the combinatorial search become intractable even for Arm Elimination? The paper tests up to ~81 combinations; production pipelines with 5+ roles and 10+ candidate models each blow past that.
  • SourceDoes the "weak planner + strong solver" pattern generalize, or is it specific to HotpotQA's delegation dynamic? Recommender-critic, drafter-editor, and retriever-generator topologies might invert. Partially answered — it inverts (2026-08-03): on Cursor's long-horizon build task the efficient frontier is the opposite assignment, strong planner + cheap worker, with the entire worker fleet costing $411 under an Opus 4.8 planner versus $9,373 when a frontier model did both jobs at the same quality. The reconciling variable is what the planner is able to do: HotpotQA's planner could answer directly and did; Cursor's cannot. So the pattern is specific to the delegation dynamic, and the general rule is about foreclosing execution at the planner role rather than about weakening the model in it.
  • SourceWhat's the right way to re-evaluate when the tool environment changes? AgentOpt assumes fixed tools — adding or removing a tool potentially invalidates the whole frontier.
  • SourceIs there a cheap per-call classifier that can predict which combination will win on a given query, avoiding combo-level evaluation entirely? Sharpened (2026-08-03): Writer's harness swap proposes classifying on feature demand rather than difficulty — which orchestration features (delegation, MCP tool use, multi-step workflows) a request will exercise — on the evidence that those features carry capability floors and that a model below the floor fails on them regardless of how simple the prompt reads. That is a candidate classifier target, not a classifier: nobody has built or evaluated one.
• Codex App Server Protocol3 open
  • SourceIs there a public schema registry so external orchestrators can target specific App Server versions without generate-json-schema?
  • SourceThe "dynamic tool calls (experimental)" caveat — what's the stability roadmap? Symphony depends on this for its security model.
  • SourceHow well does the protocol handle multi-modal turns (image inputs, screenshot attachments)? The spec is text-focused.
  • ResolvedHow does the App Server protocol compare in detail to MCP? Both expose tools to a model, but App Server is inside the Codex runtime while MCP is outside. When does each win? Answered: App Server vs MCP, and the Claude-Side Equivalent: Three Boundaries for Driving Agents — they sit at different planes and mostly compose rather than compete: MCP is the model↔world tool plane (write once, consume on every surface, provider-operated), App Server is the orchestrator↔runtime session plane (thread lifecycle, turns, events, timeouts, token accounting — none of it in MCP's scope). The only overlap is dynamic tool calls, where the rule is: MCP for reusable cross-surface third-party capabilities; orchestrator-injected tools for session-scoped, credential-sensitive ones (the linear_graphql token-isolation pattern, which also shrinks the poisoned-metadata/rug-pull attack surface to first-party code) — at the cost of experimental stability and zero ecosystem reuse.
  • ResolvedIs there an analogous protocol on the Claude side, or is Claude's equivalent exclusively the Agent SDK + tool-use API? Comparing the two would clarify when "drive an existing CLI" beats "build on the SDK." Answered: App Server vs MCP, and the Claude-Side Equivalent: Three Boundaries for Driving Agents — no documented Claude-side protocol; the offering brackets the App Server's position: claude -p (drive the product, inherit the full harness — permissions with abort-don't-hang unattended behavior, skills, context files, MCP wiring — but get text, not structured events) and the Agent SDK (build a different product on the raw runtime — Claude Design's weekend prototype). Rule: drive the CLI when the product's harness is the value and orchestration is batch/fan-out shaped; build on the SDK when the agent is a different product with its own surface and tools. Symphony's own tmux→protocol evolution marks the middle layer (structured session control over the product harness) that the Claude side currently approximates from either end — whether it gets standardized or harness shrinkage makes it moot is the watch item.
• Context Lifecycle Management6 open
  • SourceDoes the input-token reduction survive a matched billed-cost audit once side-channel planner calls and prefix-cache breaks are charged? The paper reports prompt-surface impact and explicitly declines this claim. Partially answered (2026-08-03): Prompt-Cache Economics supplies the audit for the class but not for Self-GC — CAPC reconciles measured API spend against Anthropic's invoice to within 1% and finds a token-reducing technique (query-aware compression, 3× fewer tokens) costing +40.1% more than sending nothing compressed on τ-bench retail. So the concern the question encodes is real and measured; a Self-GC-specific billed-cost audit still doesn't exist. Annotated (2026-08-04): agentic context management supplies an analytic bound on one half and explicitly declines the other. The overhead half: because each compaction pass operates on the already-compacted context plus recent turns, the number of passes grows only linearly and total cost is N·W·(1 + c/p) — a fixed multiplicative factor rather than a growing tax (~1.25× at the paper's illustrative p = 8, c = 2). If Self-GC's side-channel planner call behaves the same way, planner overhead cannot asymptotically eat the savings. The cache half is dropped by assumption ("ignore caching discounts, which shift the constants but not the asymptotics"), which is precisely the term the question is about — so this narrows the question to the cache break alone and answers none of it with a measurement.
  • NowIs the 0.3 expected-pruning break-even for immediate commit portable, or a function of one provider's cache pricing and TTL? Stated as an operating policy over one deployment's regression. Partially answered (2026-08-03): Prompt-Cache Economics settles the portability question in principle — the analogous threshold is ρ_cross(r) = (α − 1/r)/(α − β) over the write premium and read discount alone, and it moves sharply with both provider and TTL (α = 1.25 on Anthropic's 5-minute cache vs 2.0 on its 1-hour cache; OpenAI α = 1.0, β = 0.5). Read across, the number is not portable and the derivation is. Re-deriving 0.3 under that parametrization is now a synthesis over pages already in the wiki.
  • SourceHow does object-level GC compare against clear-and-restart on the same traces? No source in the corpus measures the two against each other, and they optimize different things — GC preserves in-run dependencies, clearing resets attention quality.
  • SourceHas anyone other than a vendor measured that validated compaction actually preserves fidelity where crude summarization does not? The whole three-regime argument turns on a cell — linear cost with checked fidelity — that no source in the corpus has isolated. Maximem's 92.0%/93.2% are end-to-end system scores on conversational-memory benchmarks with no compaction ablation, its validation mechanism is undisclosed, and Self-GC's no-impact judge grades candidate plans offline rather than a returned validation score. The falsifiable form: an A/B of the same compactor with the information-loss check on and off, on the same traces, scored on downstream task success.
  • SourceDoes the "no semantic interference from coordination volume" null survive at the prompt lengths agents actually run? RCWT's intact-task ablation holds at ceiling to a 0.95 coordination ratio, but its largest condition is ~14,000 total tokens — while the sessions this page's other sources measure average 70–90k input tokens per request. The falsifiable form: rerun the intact-task ablation with the same 698-token task block at 100k, 250k and 500k of surrounding coordination content, on models whose per-model effective ceilings Context Window Smart Zone shows are not predicted by the advertised window. If the null holds there, displacement is the whole story; if it breaks, there are two mechanisms and the corpus has been attributing both to one.
  • SourceWhat fraction of agent memory failures are reasoning-sufficiency failures rather than retrieval failures — the bridge document missing while a relevant document was returned? Every memory benchmark in the corpus scores a hit against a single gold target, so the quantity is structurally unmeasurable by all of them; the only bearing datum is that LongMemEval's multi-session category (75.2%) carries nearly all of one system's residual error. Named trigger: Maximem states a benchmark measuring accuracy, latency, token efficiency and context-rot resistance together is forthcoming.
• Context Window Smart Zone3 open
  • SourceDoes the smart-zone marker scale with model size, or is it bounded by attention architecture? Pocock observes "the dumb zone has become less dumb lately" but pegs it at 100K through 2026. Partially answered 2026-08-04 by prompt design at scale (empirical) — neither cleanly, because the premise of a single marker doesn't hold. Degradation onset is a per-model effective ceiling that the advertised window does not predict: two models sharing the same documented 1,000,000-token ceiling diverge sharply in format-spread growth over the identical 256k→512k range. It also splits the question by task — retrieval holds at 0.98–1.00 through 64k and past 128k for some models, well beyond the 100K marker, which is consistent with Pocock's own retrieval-vs-reasoning carve-out. The paper measures no reasoning task, so whether the ~100K reasoning marker is architectural remains untested.
  • WaitWhen sparse-attention or memory-augmented architectures ship, does the smart zone become a soft constraint?
  • SourceHow should harnesses surface remaining smart-zone budget to the user — token count, percentage, or a richer signal?
• Cost-per-Task Over Cost-per-Token3 open
  • SourceDoes "cost-per-task is lower for more intelligent models" survive measurement on non-Anthropic production traffic? The claim is stated without data and the published curves are explicitly illustrative. Still open, with a near-miss (2026-07-30): DeepMind's Gemini 3.5 Flash-Lite card is a non-Anthropic instance of the shape — +67% output price, a full agentic tier of capability — but reports no tokens-per-task, so it supplies the premise and not the measurement. Partially answered (2026-08-03), and it splits: Cursor's four model mixes are the measurement — non-Anthropic infrastructure, a real four-hour workload, matched time budgets, matched quality, published dollars. Within the planner role the claim holds: the more expensive Fable 5 planner billed slightly less than Opus 4.8 at roughly twice the per-token price, because it emitted far fewer planning tokens. At the level of the whole run it fails: the same Fable configuration came out substantially more expensive because its workers burned several times the tokens, and the most expensive run of all was the strongest model used throughout ($10,565 versus $1,339). So the thesis appears to be a claim about a role, not about a system, and no source yet measures it on a single-agent workload outside Anthropic. Sharpened, with the first counter-datum (2026-08-03): Writer's harness swap publishes per-model cost and per-model quality for six models on non-Anthropic infrastructure under one pinned price table, and cost per task rises monotonically with model strength on that workload — quality per dollar is worst for the two strongest models (Palmyra X6 3.16, Sonnet 4.6 3.27) and best for the cheapest (Qwen 3.6 4.44). It is a controlled bench rather than production traffic, the arms are not iso-quality, and the capability spread is only eight points, so it does not close the question — but the sign is wrong for the vendor guidance and the paper's own conclusion is that the model menu is the smaller lever anyway. Closest yet, and it splits again (2026-08-04): Databricks' internal coding bench — real engineering tasks on its own multi-million-line codebase, measured by neither Anthropic nor a model vendor — puts Opus 4.8 at $1.94/task and 87% success against Sonnet 5 at $2.09 and 81%, on tokens ~1.7× cheaper. Within Anthropic's own line the claim therefore holds, in the long-horizon coding regime where its mechanism should be strongest, and this is the first time it holds on a third party's real codebase. Against the wider menu it fails: open-weight GLM 5.2 is statistically tied with Opus 4.8 on quality at $1.28/task. So the surviving form of the rule is about tokens-to-completion, not about price tier — a cheaper model is dearer per task when it burns more tokens and finishes less often, which is contingent, not structural. Still not closed: case-study secondary reporting, production-derived tasks rather than production traffic, and no n, variance or per-arm methodology behind "statistically tied." The article's own cited generalization (arXiv 2603.23971 — a third of comparisons invert; Gemini 3 Flash 80% cheaper listed, 38% dearer in practice) is the paper most likely to settle this and is not yet ingested.
  • SourceDoes the advisor strategy's result (within 10% of the advisor's score at 63% of its price) generalize beyond SWE-bench Pro and the Sonnet-5/Fable-5 pairing — and where is the crossover at which advisor calls cost more than they save?
  • SourceIs "start with the strongest model" safe inside multi-role pipelines, given AgentOpt's finding that the strongest model was the worst planner? Anthropic's Sonnet-for-sub-agents note hints at a boundary it never states. Partially answered (2026-08-03): Cursor's production swarm says the safe form of the rule is positional — strongest model as planner, cheapest capable model as worker — and that running the strongest model in every role is the single most expensive way to reach the same grade. It also dissolves the apparent conflict with AgentOpt: Opus was the worst HotpotQA planner because it answered from parametric knowledge instead of delegating, and Cursor's architecture makes that impossible ("a planner never implements"). The failure is a property of harnesses that let a planner execute, not of strong models in the planner seat. Still unsettled: whether the ordering survives on tasks where the worker's job is judgment-heavy rather than instruction-following, which is the regime Cursor's own framing exempts.
• Crystallizing Agent Work into Workflows2 open
  • SourceDoes the lifecycle transfer out of IT operations? Every number here comes from incident response, which is unusually repetitive and has a crisp success oracle (the incident resolved). Coding, product and research work have neither property in the same degree. The falsifiable version: apply the Type 3→2→1 promotion criteria to an agentic coding pipeline and measure whether the deterministic share rises at all over comparable time.
  • SourceThe platform-level cost curve has no counterfactual arm. How much of the >70% per-incident cost fall is crystallization, and how much is ordinary model-price decline plus caching over the same eight months? A controlled comparison, or a decomposition against contemporaneous list prices, would separate them.
  • ResolvedCrystallization and harness shrinkage give opposite instructions at a model upgrade — delete the scaffolding versus keep the evidence-gated permissions. Which governs, and does the answer differ for instruction scaffolding versus authority scaffolding? Answered: Authority and Audit Survive Abundance — neither governs the other; they govern disjoint objects, separable by one test: can the model being better make this line unnecessary? Instruction scaffolding encodes a task prior (shrinkage governs — ablate at every release); this page's authority scaffolding encodes a boundary plus a local evidence record, neither of which a capability jump supplies — and the security corpus makes the stronger claim that authority cannot migrate inward, because a component that grants its own scope is circular ("you can delegate judgment; you cannot delegate authorization"). The sort is by what a line encodes, not where it lives (a prompt-borne scope declaration is still authority-class; the derived page grounds this in the constraint/request asymmetry). At upgrade day the reconciliation is already in this page's design: the launch pass prunes instructions, permission grants stay untouched, and the demotion circuit-breaker re-earns authority from evidence continuously — so the upgrade moment requires no authority decision at all.
• Deep Modules for Agents3 open
  • SourceHow big is "deep enough"? Pocock's example modules are several hundred LOC; Ousterhout's textbook examples are larger. There's a sweet spot; not articulated.
  • SourceFor ports/adapters codebases, does the deep-module advice transfer cleanly? The "small interface" is the port; the "large behavior" is the adapter. Probably yes, but not exercised in source.
  • SourceRefactor cost vs benefit: when is "improve-code-base-architecture" worth running on a working repo?
• Deep Research Agents4 open
  • WaitDoes the orchestration advantage shrink as base models cross the next thresholds, or is open-ended retrieval/synthesis a durable harness asset (unlike, say, prompt scaffolding)?
  • SourceDRACO grades single-turn interactions only. How much of real deep-research value is in the multi-turn loop (clarifying questions, follow-ups) that the benchmark doesn't yet measure?
  • SourceFactual accuracy is the weak axis everywhere — is the fix better retrieval, better verification-in-the-loop, or a tool-grounded check the way Lean grounds proof search? Partially answered by is deep research reliable (arXiv 2607.20891, empirical), which rules out one branch and measures a second. Not retrieval: misleading-evidence reach is already 72–98%, search-result rank moves FCAR by 1.7pp, and adding documents past the first buys nothing — retrieval is not the filter and making it better cannot be the lever. Verification-in-the-loop helps and does not suffice: pre-research verification prompting takes DeerFlow from a 60–76% baseline to 37–57%, a post-research refinement agent to 20–58%, the combination to 15–62% — and the combination is worse than either alone for Intern-S1-Pro, because the refinement step re-retrieves from the same poisoned pool. The tool-grounded branch is untouched and remains the open half: open-domain factual claims have no Lean, and the nearest thing measured here (a search-enabled verifier) is exactly what already gets these documents right in isolation and is never invoked in-workflow. So the live question narrows to where in the workflow verification must sit, not whether it helps.
  • SourceThe verification asymmetry is inferred, not isolated: the misleading corpus was selected for unanimous verifier agreement, and the standalone verifier's job (judge one document, with search tools and explicit search discipline) is strictly easier than the agent's (judge a document while executing a research task). Does an in-workflow verification step given the same tools, the same focused prompt, and its own budget close the gap — or does carrying a task degrade the check regardless of how it is prompted? The pre-research defense is the weak version of this experiment (it asks, but grants no separate step and no tools) and recovers roughly half the gap; the strong version has not been run.
• Deterministic Pre-Execution Gates3 open
  • SourceThe paper never runs the obvious baseline: does forcefully prompting the four gated rules, or adding a reflection step, recover the +12.4pp on the non-deceptive failures? The authors argue prompting cannot help where the user asserts false state, which concedes the rest. Until someone runs it, "reason less, verify more" is a claim about the deceptive slice generalized to the whole.
  • SourceHow much of the recovery is the block and how much is the rejection message? A gate returns a structured reason the agent re-plans on, so the lift may be partly informative feedback rather than prevented corruption. Falsifiable cheaply: rerun the suite returning a generic "rejected" with no reason and compare. The deterministic guarantee over the blocked write is unaffected either way; the attribution of the 12.4pp is not. Partially answered (2026-08-04) by Harness-Induced Belief Divergence (Yi & Song, arXiv 2607.04528, empirical), from the safety side rather than the success side: a block that withholds its reason leaves the disposition intact — 42 of 60 blocked destructive-command steps re-propose a same-class risky action within three steps (UnsafeRetryRate 0.700) — and gating measurably relocates the model's failure attribution onto the harness policy. That is evidence the message carries real weight, but it is not the requested attribution: that paper reports no task-success number at all, and never runs the with-reason vs without-reason contrast. The cheap experiment is still unrun.
  • SourceGate precision was audited against ground-truth trajectories, which a deployment does not have — and without that audit baggage_allowance (5% precision, 40 false blocks in 42 fires) ships silently. What deployment-time signal substitutes: post-rejection completion rate, human adjudication of a rejection sample, or a per-gate A/B? Nothing in the corpus proposes one, and a gate suite with no precision signal is a new silent failure mode wearing the old one's clothes.
• Document Parsing as the Retrieval Bottleneck2 open
  • SourceDoes the "parsing errors propagate into 7 of 12 pain points" cascade hold up under measurement rather than assertion? Layerwise Omission Attribution supplies the instrument — canary taps make an L0 loss exactly countable, and the (conflict − literal) needle contrast isolates downstream behavioral loss — so the falsifiable version is: on a fixed corpus, parse with a structure-preserving and a flat-text parser, hold every later stage constant, and attribute the delta in end-to-end failures by layer. Nothing in the corpus does this.
  • WaitHow far has the specialist-parser advantage over general frontier VLMs actually narrowed? ParseBench Fig. 5 (a vendor-run benchmark) puts LlamaParse Agentic at 84.9% against Gemini at its high setting near 76%, at roughly half the cost per page. If Harness Shrinkage as Models Improve governs here as elsewhere, the specialist layer is a temporary tax on current VLM weakness; if bbox grounding and per-page cost predictability are structural, it is not. Falsifiable at the next frontier VLM release by re-running the same harness.
  • ResolvedIs the audit-trail argument for retrieval strong enough to survive genuinely cheap long context? The deck's three reasons are cost, governance and auditability, and only cost is a function of token price. If a 1M-token window becomes ~free, does per-chunk permission filtering and citation-log auditability still force a retrieval layer — or do they become an attribution problem solvable inside the window? Answered: Authority and Audit Survive Abundance — yes, wherever the requirements it serves exist, because only the cost leg is token-priced. Governance survives by circularity: per-chunk permission filtering is per-call authorization at the retrieval boundary, and enforcement cannot live inside the window it polices ("'model promised to ignore' is not a boundary" is the in-band collapse the security corpus measured) — the window is the wrong trust domain at any price. Auditability survives because in-window attribution is model testimony where an audit needs a log produced outside the model — and a log without selection reads "everything," which attributes nothing: retrieval is the act that makes a citation log non-trivial. The scope conditions run the other way too: governance forces a retrieval layer only where principals > 1, audit only where accountability is required, and the leg this page's "~free" premise prices too generously is capability (Context Window Smart Zone's effective-ceiling/refusal evidence) — the one leg made of current model limitation rather than structure. The requirement also binds the compiled-wiki rival: compilation and retrieval are both selection-with-a-record, and corpus-stuffing is the only architecture the audit leg eliminates outright.
• Dynamic Workflows: An Algebra for Agents4 open
  • SourceThe "algebra" is still unpublished as a vocabulary. Sumner publishes the loop's shape (pop a task → implement → parallel review → apply) and its inventory (~50 loops), but no combinator names, no composition operators, and no workflow source. Partially answered: the sequence/parallel primitives are visibly in use and the model demonstrably authors and edits them from English instructions mid-run; what's missing is whether there is any structure beyond while + Promise.all. Annotated 2026-08-04: the changelog exposes the first product-surface handles on a workflow — a size guideline (workflowSizeGuideline, small/medium/large, default "fewer than 15 agents"), workflow.run_id/workflow.name OTel attributes, an agent grid, and a running-workflow status line — but still no combinator names and no workflow source. Size and observability, not vocabulary.
  • SourceIs model-authored orchestration more token-efficient than a hand-built harness for the same task? Partially answered — one side now has a number — 5.9B uncached input / 690M output / 72B cached reads / ~$165k for the Bun port — and Sumner's remark that "I would've had to write my own harness to pull this off otherwise" concedes the comparison was never run. No counterfactual harness exists, so the efficiency question is unanswerable from this source and needs a task run both ways. Complicated further (2026-07-27): the ~$165k is token cost to the merge only, excluding CI, employee time, and an ongoing post-merge tail, so even the one number on the board is not the campaign's cost. Sharpened (2026-08-03): the comparison the question asks for may be against the wrong baseline. OrchBench prices orchestration against a single serial agent rather than against a rival harness, and finds multi-agent plans consume roughly 1.5× the tokens of serial execution across every planner and context limit it tested — agent startup (1,200 tokens each), cross-agent communication, and compression overhead are unavoidable costs of fanning out at all. If that holds outside simulation, the honest form of this question is not "is model-authored orchestration cheaper" but "what does the token premium buy," and OrchBench's answer is: quality only while the working state overflows one context window, plus wall clock. The experimental design now exists, run on the wrong pair (2026-08-03): Cursor re-ran the same task under two harnesses at fixed models and a fixed time budget — the exact controlled shape this question needs — and the deliberately engineered harness reached the same grade with a fraction of the commits, conflicts and code. But both arms are hand-built, so it prices harness engineering, not model authorship. Someone now has to run this design with one model-authored arm.
  • WaitWhat did the Bun port cost to shipped rather than to green — CI, employee time, and post-merge agent spend included? Lockwood guesses ~$800k from an assumed $10k/day, which is not an observation; settling it needs either a first-party total or a public v1.4.0 release that closes the tail and dates it. The related tell is whether the ~2,475-and-rising open robobun PR queue drains (stabilization debt) or holds steady (continuous-agent-fleet throughput).
  • SourceHow far does the pattern degrade without a verification substrate? Sharpened, not answered: Bun's oracle had a property most codebases lack — the test suite was written in a different language from the implementation, so it survived the port unchanged; assertion count (1M+) is the visible variable but language-independence is the load-bearing one. What would settle it is a comparable port where the tests are written in the source language. Corroborated, still not answered (2026-08-03): Cursor's SQLite swarm is the second giant-swarm success in the corpus and its oracle has the same property in a stronger form — sqllogictest grades query results across different engines, so it is independent of the implementation entirely, not merely of its language. Two-for-two on implementation-independent oracles moves the confounder from "possibly incidental" to "possibly necessary" and supplies no negative case. The question is unchanged and now better motivated.
• Failures That Look Like Success2 open
  • SourceIs "internal state correct, final message stale" a general LLM-agent failure signature (state/utterance divergence) or an artifact of session-state architectures like ADK's? A cross-framework tally would tell.
  • SourceWhat fraction of production agent failures are silent-contract violations vs. loud errors? The 14/15 and 3/4 numbers are demo-sized; telemetry-scale data (Production-Sourced Evaluation) could ground the class. Partially answered, and reframed, 2026-08-03 by reason less verify more (empirical): 78% of observed failures on the τ²-bench airline domain are silent wrong-state failures with no tool error — benchmark-scale (250 trials over 50 tasks, replicated on 15 disjoint seeds), not production telemetry, so the question's original ask stands. What it does settle is that the question is mis-posed as a single number: the same paper's negative controls find no silent class in τ²-bench retail (self-enforcing tools raise loud errors) and zero gate firings across 200 BFCL entries. The fraction is set by whether the tool layer enforces its own preconditions, so the answerable version is "what fraction of production tool surfaces are policy-permissive?" — still unmeasured, and now the cheaper question. Explicitly not advanced 2026-08-03 by where facts go missing, despite carrying larger numbers: its 75,476-trial waterfall is a fault-injection decomposition the paper refuses four separate times to read as prevalence, and its 372-trial real-data pilot scores end-to-end non-success (0.578, or 0.509 excluding 52 execution errors) on an endpoint broader than silent omission. It does independently name a representative deployment study as the missing work — two labs now pointing at the same gap from opposite ends of the pipeline. Also explicitly not advanced 2026-08-04 by who and when pro — the corpus most likely to be mistaken for an answer, at 12,326 failed trajectories across 26 benchmarks. It cannot advance the question by construction: every trace is a deliberately injected error into a run that had already succeeded, so its failure distribution is designed rather than observed, and no trace in it was sampled from anything running in production.
• Harness Build-vs-Buy3 open
  • SourceWhat fraction of upstream churn does a narrow fork actually inherit? 13 PRs/day is the full-surface upper bound; nobody has measured the delta for a fork that drops the UI, multi-provider support, and deployment surface. A rerun restricting the diff to a realistic kept-file set would settle it.
  • WaitDoes the merged-PR rate of coding-agent harnesses peak and fall as models improve, or keep rising? Harness Shrinkage as Models Improve implies an eventual peak; the observed trajectory (Codex ~124 → ~1,000 PRs/month over 12 months) is still climbing. Re-pull the same four repos in twelve months.
  • SourceIs bug-fix share comparable across projects at all, or is the 16–68% spread purely label hygiene? Shah says compare directionally and flags Hermes' 68% himself; a uniform commit-message classifier rerun across the four repos would separate signal from convention.
• Harness Shrinkage as Models Improve2 open
  • WaitThe Boris "100 lines" prediction is a year out from May 2026 — testable in 2027. Partially answered: Harness Build-vs-Buy supplies the first measurement of harness codebase size and its trend — four comparable harnesses at 1.05M–1.75M lines with merged-PR rates still climbing (Codex ~124 → ~1,000/month over twelve months). That is evidence about the reference class, not about Claude Code, which is closed-source and unmeasured; the 2027 test stands.
  • WaitIf harness work shrinks, what new work expands to fill it? Cat Wu's bet: PM/product taste, eval-writing, character work.
  • ResolvedDoes all prompt scaffolding eventually migrate into the model, or does some remain — e.g. organization-specific style, security rules, brand voice? Answered: What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful? — no: only behavior requests migrate (and past expiry turn harmful per Instruction Compounding). Five classes survive, sorted by the bitter-lesson exemption rule (structure encoding a task prior migrates; structure encoding boundaries, records, identity, or serving arithmetic doesn't): boundary enforcement (security rules survive as constraints, the instruction class that keeps working), organization-specific record (a smarter model can't infer an unrecorded decision — org style qua arbitrary convention lives here), deliberate identity (brand voice/character is held stable across capability jumps by design), inference/deployment structure, and human-facing legibility (which grows). Communication calibration flows the opposite way — added, not removed, as defaults lengthen. "Harness shrinkage" is really request shrinkage; the prompt converges to a residue of constraints, records, identity, and calibration.
• Harness-Induced Belief Divergence3 open
  • SourceDoes harness-induced belief divergence actually cost anything? The paper's framing claim — divergence at preserved terminal success — is never measured, and the falsifiable version is cheap: report pass rate per harness alongside D_growth on the same grid. Until someone does, "the harness changes beliefs but not outcomes" and "the harness changes beliefs and outcomes and this paper cannot see which" are equally consistent with every number here.
  • SourceHow much of D_arrival is a real interface difference and how much is constraint-string vocabulary? Lemma 1 guarantees D_belief >= 0.25 from disjoint constraint sets alone, Assumption 1's canonical embedding is supposed to remove exactly that, and the observed 0.975–1.000 arrival readout says it does not. A semantic constraint matcher (embedding or entailment) in place of normalised-string Jaccard would separate the two in one re-run.
  • SourceDoes the UnsafeRetryRate result generalize past the 0.700 measured on one 15-task group with one unnamed model — and does making the block's reason visible reduce it? The paper argues blocked-action logging should, but measures logging's effect on divergence, never on retry rate. This is the same question Deterministic Pre-Execution Gates asks about the rejection message, from the safety side rather than the success side.
• Instruction Compounding3 open
  • SourceIs the review case the same mechanism as over-verification, or two? "Be conservative" reducing detection looks like high-fidelity literal instruction-following; over-verification looks like behavioral summation. Distinguishable by testing whether a weaker verification instruction still over-verifies.
  • NowAnthropic's list is hand-curated per release. Is there a detectable signal — from eval deltas, token counts, or the model's own read of its system prompt — that flags which existing prompt lines have become compounding, so pruning is not a manual reread? Partially answered: What Scaffolding Survives Model Improvement — and How Do You Know When a Line Turns Harmful? — the signature exists: a compounding line shows ablation non-inferiority (removal holds or improves quality while cutting tokens — Anthropic's own criterion, and the decisive per-line test) and inverted dose-response (escalating the instruction worsens the metric — the Unproductive Self-Verification effort-inversion fingerprint, detectable from eval deltas without ablation). Native-behavior baselining (does the uninstructed model already do it?) ranks candidates, and only request-form lines need testing (constraints don't compound). The model's own read is the one signal to avoid — naming a failure surfaces it. Residual: nothing in the corpus automates this; the per-release list is still hand-curated.
  • SourceDoes compounding require the instruction to name a behavior the model already has, or does any redundant instruction degrade output? Falsifiable by ablating one instruction at a time against a fixed eval. Partially answered 2026-08-04 by prompt design at scale (empirical) from the far side: redundancy is not required for degradation at all. Its 10-to-160 rules are distinct, non-overlapping and individually satisfiable, and all-rules compliance still floors by N=80 on all five models. So the answer to "does any redundant instruction degrade output" is bounded by a prior fact — instruction volume degrades the set regardless of redundancy, and the one-at-a-time ablation proposed here is exactly the method that cannot see it, since no single line is at fault. The redundancy half remains open and still needs the ablation.
• Knowledge-Centric Self-Improvement3 open
  • SourceDoes curation actually beat storage? The paper's conceptual separation from experience-reuse and memory systems (ExpeL, AgentKB, Agent Workflow Memory, Voyager) is argued in Related Work and never run as an arm, and no ablation removes the forums or distillation from its own protocol. A single arm — raw attempt table only, no forum, no distillation — would settle it.
  • SourceRun both self-improvement artifact classes under one protocol — an evolved harness and a distilled knowledge bundle on the same tasks, same budget, same held-out and cross-model splits. What Makes a Self-Improvement Artifact Transfer? predicts the harness gains vanish off the matched pathology while the bundle gains persist across solvers; the prediction is falsifiable and no source has run it.
  • SourceDoes the curated base keep adding value past 10 generations, or does it saturate once the easy tasks are retired? Every run stops at 10 generations with solved tasks removed from the pool, so the reported curve confounds knowledge accumulation with a shrinking task pool.
  • ResolvedWhat distinguishes a self-improvement artifact that transfers from one that does not? Answered 2026-08-04: What Makes a Self-Improvement Artifact Transfer? — an artifact transfers exactly as far as the regularity it encodes extends. The fitted-to-task vs fitted-to-model binary is the measured special case: harness patches encode solver pathologies (transfer within the pathology class — Qwen 27B→397B +11.0 nearly loss-free, Gemini +13.5 across families on the shared pathology — and fail outside it), distilled insights encode domain regularities (reuse holds the domain fixed, so all eight cells survive solver churn). Cross-release instruction depreciation is the same phenomenon on the time axis (the synthesis carries the evidence), transferability can be enforced at write time (this paper's schemas are the mechanism), and what transfers when the artifact doesn't is the procedure. The controlled two-artifact test remains open above.
• Latent vs. Deterministic Space2 open
  • SourceTan asserts the "wrong side" diagnosis covers most AI-engineering bugs. Does any incident/failure taxonomy (agent postmortems, eval failure analyses) actually classify failures by computation-locus, and what fraction lands in each side? Partially answered 2026-08-03 by reason less verify more (empirical): the first failure analysis in the corpus that classifies by locus and attaches a fraction — on the τ²-bench airline domain, 78% of observed failures are silent wrong-state failures traceable to policy living in a prompt document rather than in the tool, and moving four rules to the deterministic side recovers +12.4pp. Three limits keep it partial: it is one benchmark domain, it classifies along one axis (policy compliance) rather than taxonomizing failures generally, and the paper's own negative controls show the fraction is set by how the tool layer was built, so it is not a population estimate for agent bugs at large. Nothing yet measures the other direction of the diagnosis — code hard-coding judgment that belonged in the model. Advanced further 2026-08-03 by Layerwise Omission Attribution (where facts go missing, empirical), which supplies the taxonomy half almost completely: nine layers covering the whole pipeline, split explicitly into deterministic software (L0-L3) and model behavior (L4-L8), with a waterfall that assigns every lost fact to exactly one locus and a fixed order preventing double-counting. Its answer to the fraction half is 73.4% software — but that number comes from an allocation of deliberately injected faults, which the paper fences four separate times, so the taxonomy transfers and the fraction does not. Still open: any locus split measured on organic incidents, and still nothing on the reverse direction.
  • WaitThe seating example prices latent-space judgment at "a couple hundred dollars of tokens" for 800 seat assignments. As models absorb more deterministic capability (Harness Shrinkage as Models Improve), does the economically-optimal boundary move toward latent space, or does state-out-of-context remain invariant?
• Layerwise Omission Attribution3 open
  • SourceUnder organic production faults rather than Phase A's deliberate injection, does the L0-L3 software share stay above the L4-L8 model share at all? The 73.4% figure is a property of the designed cell allocation and the paper fences it four times; nothing in the corpus measures the layer split on unweighted production traffic. Distinct from Failures That Look Like Success's prevalence question, which asks what fraction of failures are silent rather than which layer produced them.
  • SourceDoes ω4 = 0.000 survive a workload that forces real context overflow? Engine/memory contributed zero attributed loss while q4 KV cache carried OR 2.25, because L4 is credited only when an event log shows the needle sat in an evicted range and corruption of a resident token scores at L5. Cheaply falsifiable: rerun the B1/B2 contrasts with payloads sized to evict the needle span and check whether the L4/L5 split moves. If it does not, the zero is an artifact of the attribution rule and the taxonomy needs a fifth deterministic class for corrupted-but-resident tokens.
  • SourceDo the L0-L3 checkpoint taps transfer to a hosted-API agent pipeline? The deterministic half of the method depends on reading raw bytes and token IDs at every boundary — T4 requires the final token-ID sequence, T5 requires KV-residency from engine logs — and no frontier inference API exposes either. If they do not transfer, the exactly-countable layers are auditable only on-prem, and the stack most agents actually run on is the one that cannot be instrumented.
• LLM-as-Compiler Knowledge Base3 open
  • SourceAt what scale does the no-vector-database approach break down? Karpathy's ~100 articles fit in context, but what about 1,000+?
  • SourceWhat's the optimal granularity for concept articles — one concept per article, or clustered by theme? Partially answered (2026-08-03): Knowledge-Centric Self-Improvement answers a machine-read version of this and its answer is neither — granularity is carried by scope conditions attached to each claim (applies_when / does_not_apply_when) rather than by article size, with two levels of store (per-task and cross-task) whose inputs are kept disjoint. It also supplies a measured caution: transferring a fixed quantity of knowledge made recipient memory "noisy or detrimental," so their adapter bounds delivery at 0-3 items per field and returns empty lists when the prior is weakly relevant. Suggestive, not settling — bundles consumed by a solver agent are not articles read by a human.
  • SourceHow effective is the synthetic training data → fine-tuning pipeline in practice?
  • ResolvedHow to handle conflicting information across sources during compilation? Answered: When Knowledge Layers Disagree: Context Files vs Memory, and Conflicting Sources at Compile Time — a five-step protocol extracted from this vault's own practice and worked cases: (1) align constructs before declaring conflict (most contradictions dissolve into metric/population/time-axis/unit non-comparability — the Faros-vs-CMU worked example); (2) attach provenance and evidence tier, weigh by method and incentive, never average; (3) stage genuine conflicts explicitly on every affected page, bidirectionally — silent choice is the compile-time form of laundering; (4) convert staged conflicts into tracked open questions with named resolution conditions; (5) resolve at compile/lint time (Tan's librarian), so queries inherit the staged conflict with weights visible rather than re-adjudicating per query.
• Loop Engineering3 open
  • SourceOsmani's cost caveat is unquantified: at what token budget does a continuously-running loop stop paying for itself, and how do you instrument that? (Cf. Agent Loop Pattern's "who owns the budget when the model schedules its own loops.") Partially answered (magnitude only): a PostHog engineer running review/CI loops puts it at ~60% of personal token spend with no regret (Risk-Tiered Auto-Approval, case-study) — which establishes that the share can be a majority and still be judged worth it, but is a self-assessed share, not a break-even threshold or an instrument for finding one.
  • SourceIf /goal's stop-check is itself a model, what verifies the verifier? The maker/checker split pushes the trust problem up a level, not away. Partially answered (2026-08-04) — the regress is bounded rather than closed: Wu et al. (empirical) don't verify the verifier; they make its unreliability a measurable scalar (Youden's J = 1 − ρ₀ − ρ₁) and show what each level of it permits. Above roughly J ≥ 0.18 a loop can act on a calibrated estimate of the checker's noise and stay within 2.8pp of the true-parameter reference; at J = 0.03 the label-free calibration collapses (and collapses further with more data), and the correct move is to stop trusting any estimate and fall back to an estimation-free keep-best rule. What is still open is the part the question actually asks: measuring J at all needs a small labeled probe, so the trust problem is relocated to a human-labeled sample rather than dissolved.
  • WaitDoes loop-engineering converge on a single dominant shape (morning-triage → worktree → maker/checker → PR), or proliferate into many idiom-specific loops? The essay describes one shape "I keep using" but claims the primitives are general.
• MCP and Computer Use3 open
  • WaitThe MCP ecosystem's growth rate vs. computer use's quality curve: at what point does computer use become good enough that the marginal value of building an MCP server drops? Boris implies this is years off but doesn't quantify.
  • WaitIs computer use a sustainable interface or a transition technology? If most knowledge-work software adds MCP support in the next 24 months, computer use's role shrinks to legacy/desktop-only systems.
  • SourceMCP security model: as the playbook prescribes wiring MCP into Salesforce, Gmail, Calendar for solo founders, the attack surface scales with adoption. Partially answered by Zero Trust for AI Agents (tool poisoning, rug pulls, the first in-the-wild malicious MCP server) — see "MCP as a security surface" above. Open residual: how does a solo founder realistically run/host and self-sign every MCP server the framework recommends, given that the appeal of MCP was zero-integration-effort? Sharpened by ShareLock: the cheaper alternative to self-hosting — scan the tool descriptions with a guard model — is information-theoretically defeated by threshold fragmentation, so the lightweight mitigation doesn't hold and the burden falls back on run-your-own-server or downstream action-layer authorization. And Agentjacking shows run-your-own-server itself isn't sufficient: when the server is a legitimate observability platform relaying attacker-injected data (fake Sentry errors), self-hosting/vetting the server catches nothing — the untrusted input rides in on its data, so the residual burden falls squarely on the downstream data/action layer (provenance tracking + an out-of-band action gate), not on server hygiene.
  • ResolvedHow does Cowork's computer-use guardrail compare to Claude Code's auto-mode classifier? Different deployment context, possibly different risk profile. Answered: Classifier Gates vs OS Sandboxing: The Defense-in-Depth Story for Auto Mode and Cowork — same mechanism, inverted role. Cowork's guardrail is auto-mode-style classifier gating on the browser/computer-use surface (the Opus 5 card's browser row is measured on the Cowork harness: 31.5% bare → 3.70% → 0/129 scenarios with auto mode). The risk-profile difference is which layer can be load-bearing: Claude Code's blast surface is local and containable, so the sandbox can be primary and the classifier a convenience; Cowork drives the user's authenticated live SaaS sessions, where no sandbox equivalent exists and actions are less reversible — so the classifier carries the defense alone on the surface with the worst bare-model injection rate. Caveats: vendor-measured on a bounded suite, still a model-based gate (the D2 critique and the ADI forged-data failure shape apply), and the deterministic out-of-band action gate the research points to exists for neither surface yet.
• Open-Ended Discovery Harnesses3 open
  • SourceDoes the EvoX margin survive a matched budget? EvoX ran 100 iterations at ~$23.50/task against SwarmResearch's $50, and that is where the large wins are (2.635996 vs 2.1064 on circle packing; roughly doubled heuristics scores), while the budget-matched CORAL comparison is close enough that the authors call three of the wins polishable. Falsifiable directly: rerun EvoX to a $50 cap, on GPT-class models where its authors tuned it.
  • SourceThe architecture's structural half (fresh context, branch isolation) and its behavioral half (an LLM steering a population) are never separated. §3.5 reports the Shepherd defaulting to near-greedy and prescribing ideas against its own guardrail, and the Table 2 harness deliberately strips the behavioral half to a branch field and still wins — which suggests branch-preserving parent selection may be doing most of the work. Falsifiable: run the full 15-task benchmark with a random or heuristic parent-selector in the Shepherd's place.
  • SourceEvery method sits at 34–73% of human SOTA on all five contest-heuristics tasks, and the gap does not narrow with the better harness. Is that a search-procedure ceiling, or is ALE-Bench's Elo-like metric compressing a smaller objective gap into a large-looking rating gap? The paper raises the metric caveat and does not resolve it.
• Optimizer–Evaluator Decoupling3 open
  • SourceSingle-bit feedback bounds the leak per round, but not across rounds — a long enough accept/reject sequence is itself a channel into the sealed audit. Does SEAL's advantage survive a horizon much longer than the ten rounds tested (the one extended trace runs 21), or does acceptance rate and deployment truth decay as the agent accumulates bits?
  • SourceHow much independence is enough — different model family, different vendor, different modality of check (model judge vs. compiled test vs. production telemetry)? Partially answered (prescription, not measurement): PostHog (case-study) deploys the maximal-independence answer on all three axes at once — different instructions, different goals, and different models and providers per reviewer — on the stated rationale that agents are "unaware of their own blind spots." No ablation accompanies it, so it records what a practiced team judged necessary, not what is sufficient. Partially answered (measurement, one axis): HarnessBank ablates the modality axis — a deterministic evaluator plus a paired significance test, against the same loop crediting on mean improvement — and finds the difference shows up in archive quality and termination rather than in what ships (above). The model-family axis for the grader is untouched there, because its grader is not a model. Third axis, measured: disclosure. Guo et al.'s leaky-anchor arm holds the grader fixed and varies only whether its numeric scores are shown after a rejection — SEAL is at least as high in all six rows, strictly higher in five, with a 35.1 → 12.7 reversal on the worst cell. So independence is not one quantity: an equally independent grader is worth measurably less when the optimizer can read its numbers. Fourth axis, named but not measured (2026-08-03): Cursor varies the reviewer's evidence scope — full worker transcript, output only, or nothing but the codebase — alongside model, training and personality, and reports the design rule rather than the numbers: no single lens catches everything, decorrelated lenses stack. That converts the question from "how much independence" to "how uncorrelated are the failures," which is a set property and cannot be answered by grading one reviewer. Settling it needs per-lens catch rates on a shared bug set — the measurement neither production account has published. Fifth axis, measured, and it dominates the other four (2026-08-04): Zhou varies the grader's independence from the artifact rather than from the optimizer — commit an answer before conditioning on the candidate, or don't — and gets FPR 0.719 → 0.012 and discrimination 0.06 → 0.96 on identical text, with model family, scale and candidate-visibility all held fixed. That reorders the question's premises: on this task the axes this page has been sweeping buy less than the one it had not named, and the decorrelation hope takes a direct hit (three-family unanimous-accept ensemble still passes 55%; Proposition 2 rules out every monotone aggregation rule over a shared plausibility signal). Scope: an exact-matchable final answer is what makes the commitment checkable, so the result covers graders that can solve the task, not open-ended rubric grading. Sixth axis, measured in production but confounded (2026-08-04): Leni swaps the observe/compare stage of a live loop between a ~4B post-trained verifier and the frontier model that generated the artifact, and reports rescues 6 → 2 and correct rejection −4–5 pp. It is the first production measurement here, and the first where the grader sits inside the loop rather than after it — but it moves model family, model size, and post-training objective together, so it cannot say whether the effect is independence or specialisation, and the paper names the missing arm itself (an independent frontier model from a different provider). Single internal runs, vendor-evaluating-itself, two of four specialists. It sharpens the question's shape rather than its answer: "how much independence is enough" now has to be asked jointly with "how much of the observed benefit was never independence at all." The lineage axis, measured at last (2026-08-12), and it is the one this page had flagged as a hole rather than an axis: Greptile (case-study) varies only whether the grader shares the author's model family — harness, diff and ground truth held fixed — and finds each frontier model catches 6–12 fewer points of high-severity bugs in its own family's code, as a clean crossover with near-zero reviewer and dataset main effects. That answers the sub-question every prior entry deferred (different model family: yes, worth 6–12 points of recall) and reframes the Bun campaign's maximal-lineage configuration from a noted omission into a measurable cost. Three limits keep it from closing the bullet: it is a vendor's own labelled set with no released artifact and no judge validation, the effect is measured on code review rather than on optimizer-loop crediting, and it says nothing about vendor-versus-family granularity or about whether an open-weight third party sits inside or outside the cross-model band.
  • SourceThe seventh axis, unmeasured: a learned surrogate of an exogenous oracle. Jeff Dean (practitioner-opinion) prescribes replacing slow validators with neural approximations trained on the real simulator's output — a ~300,000× speedup at "nearly as accurate" for density functional theory — as the way to make automated experiment loops fast enough to matter (Recursive Self-Improvement). The surrogate is genuinely exogenous in provenance (trained from the oracle, not authored by the optimizer) but is an approximation with an error surface, and a loop running 10⁵ rounds against it optimizes that surface as readily as the objective. Where does a distilled oracle sit on this page's independence axes, and how many rounds does "nearly as accurate" survive? Nothing in the corpus measures it.
  • ResolvedDoes decoupling need to extend upstream to metric design? An optimizer that authors its own rubric has a subtler channel to game than one that merely reads scores. Answered (2026-08-03) by self authored verification unreliable (empirical): yes, and the question's framing was too generous. Guo et al. hand the optimizer both the policy and the test file and measure the divergence against a sealed deployment evaluation across six models and three seeds — 35 of 35 runs end with a self-score above 0.70 while 15 of 35 score below their game's random reference, and the per-model gap on Breakout reaches +0.92. The channel is not subtler gaming; the paper shows it is not gaming at all ("this does not require explicit cheating"), so an optimizer with a clean conscience produces the same divergence. Constraints that stay inside the self-authored instrument (monotone, discriminative) fall below no protection at all for four of six models, and the information limit α + β ≥ 1 - TV(P+, P-) says why: no endogenous-only gate can make both errors small once the regressing and non-regressing worlds look alike from inside. The sufficient fix is one sealed exogenous acceptance bit (SEAL), not honesty. Scope caveat: the instrument here is an executable test suite over programmatic Atari policies, so the result covers metrics the agent authors and runs; an LLM-judged quality rubric is the untested neighbouring case.
• Orchestration Sets Token Economics3 open
  • SourceDoes the effect survive against a competent third-party baseline rather than a vendor's own frozen loop? Every magnitude here is one harness versus one deliberately-superseded predecessor from the same company. A configuration-level cross-harness measurement of the token dimension — the paper names this as natural future work — would separate "good harness" from "bad baseline." Partially answered (2026-08-04): Databricks' internal coding bench compares three shipped third-party harnesses — Claude Code, Codex, Pi — with the model held constant, and reports the same success rate at "2x less cost" for the minimal harness plus a 3.13× per-task context gap on Opus 4.8. No arm is a superseded baseline, so the strawman risk does not apply to it. But it is secondary reporting with no task counts, no variance and no per-arm quality table, and it measures context rather than billed cost for the harness comparison — so the direction survives a third-party test while the configuration-level token measurement still does not exist in the corpus.
  • SourceDoes harness leverage hold outside the narrow band it was fitted on? r = 0.99 spans baseline capability means of just 0.710–0.789 over six models on one vendor's task set; the interesting question is whether the slope flattens, steepens, or inverts at frontier capability, where the harness is competing against a model that can increasingly do the orchestration itself.
  • SourceDoes the "orchestration beats model choice as a cost lever" ordering survive on long-horizon coding workloads? The task set mirrors an enterprise assistant (grounding, workflows, tools, content) and the paper concedes results may differ on SWE-bench-class work, where turn counts are far higher and the quadratic history term should favor the harness more, not less. Partially answered (2026-08-04): Databricks' bench is that workload — real engineering tasks against a multi-million-line codebase — and the ordering holds directionally, with harness choice moving cost ~2× at equal success while the model menu in the same write-up spans $1.28–$2.09 per task (≈1.6×). The two spreads are not measured under matched conditions: The Register never says which harness the per-task dollar figures were run under, and the per-task context figures come from a different pair of arms than the 2× cost claim. Corroboration, not replication.
• Output Length Calibration3 open
  • SourceDoes the end-of-prompt reminder work because of position (closest to generation) or repetition (stated twice)? The guide prescribes pairing both and does not separate the effects — testable by ablating the top-level instruction. Partially answered 2026-08-04 by prompt design at scale (empirical, five models) on the position half only: moving an identical, un-repeated instruction block between the system prompt and the user turn changes adherence by up to 8.7pp at N=160 — a larger effect than prompt format for four of five models — so position alone is a real lever, not an artifact of restatement. Two limits on how far that carries: the paper's placement arms are single-placement (never both slots at once), so it cannot separate position from repetition in the paired configuration this question asks about; and the direction is model-specific (user-turn placement helped two models, hurt two, and did nothing for a fifth), so "closer to generation is better" is not a rule the corpus can assert.
  • SourceDoes an explicit conciseness instruction cost quality on tasks whose answer genuinely needs length, or does the model still finish the work and only cut padding? Anthropic asserts the latter for the verification case but not for this one.
  • WaitIf the next model ships better-calibrated defaults, today's conciseness instructions become tomorrow's compounding instructions (Instruction Compounding) — does length calibration go stale the way verification instructions did, and is there a way to write it so it degrades gracefully?
• Parallel Agent Orchestration5 open
  • Waitp99 OpenAI runtime of 71 agent-hours/day is a frontier preview inside an unusually favorable environment. Does external concurrency actually trend toward it as frictions fall, or is heavy parallelism specific to model-adjacent work?
  • SourceSummed-overlap runtime can exceed 24h/day — it measures agent effort, not human attention. What is the human's actual oversight load per concurrent agent, and where does it saturate (AI Brain Fry)? Sharpened: HAS-Bench reframes the shape rather than measuring the load — in a controlled (LLM-simulated) benchmark the value of human input is configuration-dependent and non-monotonic (a right-timing / right-channel / right-authority sweet spot; more agency brings diminishing and sometimes negative returns), so per-agent oversight is likely a returns-curve with a peak, not a linear cost that hits a wall — but it is single-human, single-task, so it does not measure real concurrent-oversight load.
  • SourceConcurrency is measured over one week. Is 5+-agent management a stable practice or a burst around specific large tasks?
  • SourceDoes real coordination content displace task budget the way a synthetic template does? RCWT's block is one hand-written mix of role/protocol text, agent messages, shared propositions and tool schemas, and it topically overlaps several of the facts it scores — while the coordination traffic this page documents is dense tool dumps, verbose transcripts, design docs and contradictory agent claims. The falsifiable form: rerun the fixed-budget sweep with the coordination block drawn from real multi-agent traces (Bun's worktree shards, Cursor's swarm), varying content type independently of token count, and check whether the cliff still lands at the same residual reserve. If the reserve is content-dependent, "measure your task's residual budget" is not yet a portable rule.
  • WaitClaude Code's shipped fan-out defaults (200 spawns/session, 20 concurrent, fewer than 15 workflow agents, depth 3) are stated with no rationale and no measurement. Do they correspond to anything Anthropic measured — a runaway-loop incidence curve, a quality-vs-agent-count sweep — or are they round numbers chosen to bound a pathology? Until that is answered the product/research convergence on this page stays convergence. (Trigger: Anthropic publishing telemetry or a rationale for the caps, or a third party sweeping agent count on the same harness the way OrchBench did in simulation.)
• Prompt-Cache Economics3 open
  • SourceDoes the two-tier step survive at production prefix sizes, or is ρ ≈ 0.85 the real steady state everywhere above the threshold? §3 measures ρ = 1.0 at 4k–8k tokens while the 94k and 262k workloads both converge to ~85%, and the "multi-server replication" mechanism invoked to reconcile them is never actually characterized.
  • SourceIs the implicit caching of large tools= arrays a documented, stable provider behavior or an artifact of one routing configuration? It materially changes what explicit cache_control is worth on tool-heavy agents, and was found by accident rather than by design.
  • SourceAt what mutated-fraction of the cached prefix does query-aware compression cross from saving to costing? The two production workloads bracket the axis at roughly 15% (saves 31%) and ~50% (costs 40.1%), but no source measures the curve between them.
• Repository Exploration Subagent5 open
  • SourceDoes the gain survive better main models? The same-model-exploration result suggests the architectural benefit is somewhat model-independent, but the trained-explorer margin may erode as frontier models get cheaper and better at staying in their smart zone unaided. The bitter-lesson question is unresolved.
  • SourcePrune vs. don't-pollute. SWE-Pruner removes context after the fact; FastContext avoids accumulating it. Are these complementary (prune the solver and delegate exploration) or substitutes? Not tested together. Sharpened (2026-08-03): Tool-Output Pruning moves pruning to the agent-environment boundary, so the two now differ by one turn of exposure rather than by accumulation, and both target the same exploratory reads — which makes overlap, not additivity, the default expectation. Still measured on no benchmark together.
  • SourceHow small can the explorer go? The authors flag 1.7B / 0.6B as future work — if the recipe holds, the explorer becomes nearly free and the architecture dominates.
  • SourceGenerality beyond Mini-SWE-Agent. Only one (deliberately minimal) main-agent scaffold is tested; richer harnesses with their own memory/subagent orchestration may already capture part of the benefit or interact differently.
  • SourcePatch-derived reward leakage. Training the explorer's reward on the gold patch's file/line ranges risks overfitting to where fixes landed rather than where evidence lives; the F1-vs-recall behavior partly mitigates this, but the proxy is imperfect.
• Shared Harness, Differentiated Surfaces2 open
  • WaitDoes one harness with UX-layer differentiation stay viable as audience breadth grows, or does the abstraction tax (hidden sub-agents, 32 model options, Ultra buried in advanced settings) eventually force a re-split?
  • SourceAnthropic ships two products split by output type while OpenAI ships one merged surface — is that a durable architectural disagreement, or is Anthropic's split a May-2026 artifact that later sources show closing?
• Stopping Under a Noisy Verifier3 open
  • SourceThe damage probabilities that drive every result here (β = 0.615–0.938) come mostly from a deliberately corrupted repairer premise. What are α and β on unperturbed production loops — code repair against a real test suite, tool-call repair against an executor — where nobody injected anything? MATH-500 and BFCL-single decline without the injection, so the direction is not purely constructed, but no field measurement of these two parameters exists. Partially answered (2026-08-04): Leni's production instrumentation supplies the first one — a shipping spreadsheet recalculation loop measures c = 0.20, r = 0.75, f = 0/357 (≲1% at 95%), i.e. α ≈ 0.15 and a damage term β ≈ 0, and the loop is net-positive (+1.5 pp) with no stopping rule at all. So the harmful-repair regime is not the default outside a corrupted-premise construction. Three reasons this doesn't retire the question: the oracle is a deterministic re-execution engine (this page's own limits note that deterministic verifiers break the repeated-query estimator, so it is the easy corner); b has no field value precisely because f = 0 left no false alarm to break anything; and the loop caps at two iterations, so round-index decline is untested. The asked-for cases — LLM-judged code repair, tool-call repair against an executor — remain unmeasured, and this is a single vendor measuring its own system.
  • SourceDiagnosing "my verifier's J is too low to steer on" currently needs labels: the paper deliberately uses a held-out labeled separation test rather than the binomial-mixture EM, to avoid diagnosing a broken estimator with its own output — and the EM is precisely what degenerates as J → 0. Is there a label-free J diagnostic that stays honest at low J, or is a small labeled probe irreducible?
  • SourceThe rule is one-step myopic and lands 17pp below post-hoc round-2 selection on the non-stationary trace. Does a round-dependent (α_t, β_t) model recover interior peaks, or is the peak only locatable in hindsight because the mechanism change that creates it is unobservable at the time? The paper names this as future work.
• Ticket-Driven Agent Orchestration5 open
  • SourceWhat's the right granularity for ticket size when the unit is "what one agent does in one workspace"? The post implies "much larger units of work" become viable, but how does that interact with the agent.max_turns limit (default 20)?
  • SourceHow do you prevent a ticket-extension cascade when agents file follow-up tickets liberally? Is the only governance check human triage at the Todo-state queue?
  • SourceDoes this pattern generalize to non-software work (research, ops, content)? The DAG dependency model and prompt-as-policy file should transfer; the per-issue workspace doesn't obviously.
  • SourceWhen an agent gets a ticket "completely wrong" (mentioned in the post), how is the lesson fed back into the system? Symphony's answer is "add guardrails and skills" — what's the institutional process for that?
  • SourceHow does ticket-driven orchestration interact with sprint planning / OKRs / roadmap work that operates on aggregates of tickets? Does the abstraction collapse when tickets are scoped that small?
• Tool-Output Pruning3 open
  • SourceDoes an in-backbone pruner survive a billed-cost audit? It defeats the prefix cache over every tool-response span by design, substitutes a modified history that invalidates the suffix each turn, and raised API calls on both backbones while raising per-trajectory input tokens on one. The paper reports tokens and wall time and never money; Prompt-Cache Economics shows a 3× token reduction costing +40.1%.
  • SourceWhy does the same pruning head raise MiMo-V2-Flash's SWE-Bench resolve rate by 3.8 points and lower Qwen3-Coder-Next's by 1.2, when it helps both on the read-only benchmarks? With n = 2 backbones and no proposed mechanism, "pruning helps patch generation" is not yet a claim — a third backbone would settle whether the split tracks model scale, attention architecture, or agent-training recipe.
  • SourceIs any per-line label-match metric usable for selecting a pruner? F1 inverted against the judge on both examined cases, and the paper's fallback is an LLM judge that is itself unvalidated in this role. Nobody has checked whether the inversion also holds against a judge-free downstream metric (resolve rate on a matched harness), which would decide whether the judge is measuring usability or its own preferences.