AI Coding Practice, in order.

Open questions 83 open

• Acceleration Whiplash3 open
  • SourceFaros's own deferred question: do the bug/incident increases persist when normalized for PR size, or do larger PRs account for most of the quality deterioration? (If the latter, hard PR-size limits are the highest-leverage fix.) Partially answered by Security Debt of Agent-Generated Code (empirical, non-vendor): on the security axis, PR-level flagging rises monotonically with change size — 16.2% for 1–9-line PRs to 53.6% for 1000+-line PRs, a 37.4-point spread — which is the size-stratified evidence this question asks for and supports hard PR-size limits as a real lever. Two gaps keep it open: it measures smells introduced, not the bugs and incidents Faros counts, and being a cross-sectional association it can't say whether capping size lowers density or merely re-partitions the same changes across more PRs. Further partially answered 2026-08-12 by Tran et al. (empirical, with a human control cohort): AI changes there are indeed larger (median 89 lines vs 33, 3 files vs 2), and the downstream comparisons are stratified on change size among other covariates — so the ratios that survive stratification are not the size effect. What survives is split by outcome: blocking threads 1.92x and build failures ~1.3x stay above parity, revert rate ~0.9x stays below. So size does not account for the deterioration, and the deterioration does not have one sign. A hard PR-size limit therefore addresses review burden rather than production stability, which is a narrower case for the lever than this bullet originally assumed.
  • SourceCode churn +861% is genuinely ambiguous (Faros lists three explanations: rework of AI code, productive legacy refactoring, or accelerated polish). The cross-customer metric can't resolve it — a real gap, not a finding. A partial proxy added 2026-07-29 by circleci q2 pulse 2026 (vendor-claim): CircleCI's Merge Efficiency Ratio — validation cycles a feature branch needs before it lands on main (median 3.9, top-5% 2.6, elite cohort 1.3) — counts a pre-merge form of the same rework, and it is countable per team rather than pooled cross-customer. It narrows the ambiguity from one side only: cycles spent failing validation before merge are hard to read as "productive legacy refactoring," so a high MER is closer to unambiguous rework than churn is. It does not decompose Faros's metric, because the two measure different things — MER counts attempts, churn counts lines-deleted-to-added, and a clean refactor that passes CI first try is invisible to MER while dominating churn.
  • How much of the "maturity doesn't protect" claim survives the vendor incentive to argue exactly that (i.e., "your existing practices won't save you — you need our platform")? Partially answered by Review as the Control Point (non-vendor, empirical): its whole thesis is the opposite — AI doesn't fix the sign; team expertise and process do — which leans toward DORA's "foundations protect you" and against Faros's determinism. But it argues the moderators exist rather than measuring a maturity effect, so the vendor-incentive question isn't closed, only counterweighted by a non-vendor source that disagrees with the framing.
  • ResolvedFaros reads under-review as a widening crisis; CMU's non-vendor GitHub telemetry finds the agent no-review rate converging toward the human baseline (>50%→~14%) as orgs learn to review agent code. Is the divergence real (enterprise vs open-source populations, adoption-depth cross-section vs calendar-time trend) or does the whiplash's under-review pressure only surface where PR volume is highest? Answered: The Under-Review Divergence: Faros's Widening Crisis vs. CMU's Convergence — mostly not real: Faros's +31.3% is a delta in unreviewed-PR count across adoption depth (enterprise, all PRs) while CMU's is a falling share of unreviewed agent PRs over calendar time (open source) — a falling rate and a rising count coexist under Faros's own volume growth. The volume clause is supported (median per-project no-review ≈0% vs pooled >50%; triage by PR type), and Faros's own risk-tiered-gating remediation is the triage behavior CMU observes emerging. The residual disagreement is a forecast: does triage discipline survive agentic authoring crossing from <1% to double digits — untested in both datasets.
• Agent Review Comment Resolution3 open
  • SourceTwo empirical studies a year apart disagree by roughly a factor of two on the central quantity: Goldman et al. (ASE 2025) report 60-70% of LLM-generated review comments unresolved, this study reports 71.4% resolved. Is the gap population (industrial vs open-source GitHub), product (in-house pipeline vs shipped agents with one-click suggestion blocks), or construct (whatever Goldman counted vs a GitHub thread flag that this paper's own card sort shows under-counts by ~24%)? Ingesting Goldman would settle it; averaging the two would not.
  • SourceThe 470-discussion taxonomy is drawn only from unresolved comments that received a reply — 6.7% of the unresolved population. What explains the silent 93.3%? Reviewer fatigue, comment volume, triage, or the same context error just not worth arguing about? The taxonomy's shape (only 11 of 470 dismissed as low-value) would look very different if noise dominated the silent majority, and the two hypotheses are distinguishable by sampling the silent set directly.
  • SourceResolution is adoption, not correctness. Does a resolved agent comment correspond to a defect that would otherwise have shipped — and does an agent reviewer change any downstream outcome (escaped defects, incidents, revert rate) against a no-agent-reviewer baseline? No study in the corpus has run this on either side of the review loop; it is the same missing outcome measurement that leaves Risk-Tiered Auto-Approval's throughput figures unattached to safety.
• Agent-Generated Test Quality4 open
  • SourceThe paper's own two-stage protocol was never completed: does the 0.44 vs 0.30 candidate-rate gap survive dynamic confirmation, or do agent tests contain flakiness indicators without being measurably flakier under repeated runs? The specified experiment (1,000 sampled tests × 100 runs per cohort) would settle it directly.
  • SourceDoes the edge-case-breadth advantage survive data-flow analysis? The literal-only detector may be measuring "agents pass literals where humans pass fixtures" rather than a real coverage gap — a re-run with variable resolution, or a matched pytest-aware parser, is the discriminator.
  • SourceWhat is the survival rate of agent-authored tests? The paper's own future work names the missing quantity: how often agent tests are deleted, rewritten, or @skip-marked over subsequent months. Coverage breadth bought at the cost of a suite people learn to ignore is negative value, and nothing here measures the maintenance side. Not answered, but the first deletion figure lands nearby (2026-08-12): Dipongkor et al. find that in non-improving Java Code+Tests PRs agents delete more tests than they add — 82 deleted against 31 added, 2.6×, with a further 51.2% editing only existing test bodies. That is the opposite direction of this bullet (agents removing pre-existing tests within a single PR, not agent tests decaying over months) and it comes from 64 Java PRs, but it is the corpus's first measurement of agentic test deletion in any form, and it makes the longitudinal version cheaper to ask: the same repositories already carry the history.
  • SourceNeither AIDev study has a human baseline, and the missing comparison is now the same one twice. Agents include a test change in 49.6% of code-touching PRs and their tests raise diff coverage in 22.5–35.9% of Code+Tests PRs — but nothing establishes whether human-authored PRs in the same repositories do better. Diff coverage is computable retroactively from any merged PR, so a matched human cohort over the same 44 instrumented repos is a tractable study rather than a wish, and it would settle simultaneously whether the 0.62-vs-0.32 edge-case gap here survives a targeting-aware metric.
• Agentic Coding Work-Composition Shift3 open
  • SourceThe window is seven months and the value proxy is coarse/relative. How much of the +27% is genuine task-complexity growth vs. classifier/marketplace-matching drift?
  • SourceThe study excludes headless/SDK/IDE usage — a "substantial share," and likely the most automated/end-to-end. Does including it accelerate or reverse the composition shift?
  • SourceIf "fixing" keeps falling, is that because models break less, or because broken-code work is migrating to non-interactive pipelines this study doesn't see?
• Agentic Work Systematization4 open
  • WaitThe 5.4%→26.6% curve is three months. Is this a durable behavior change or a novelty spike following a Codex skills-feature push? (Cf. the OpenAI-internal training campaigns the paper notes.)
  • SourceCustom skills encode org-specific context — but who maintains them as the codebase and conventions drift? Systematization could itself become a debt surface (Agentic Technical Debt) if skills rot. Partially answered: skill registry to repository lifecycle (empirical, 2026-07) measures the drift directly on public GitHub — 53% of reused skills are never modified after adoption, 40.2% of never-updated copies sit on a changed upstream, and local maintenance is 2.7:1 additive (6.1:1 for locally authored skills), with rename/tooling-substitution chasing the single largest evolution activity (24.3%). So skills do rot and the ratchet is real. Still open: the consequence side — no study yet links skill staleness to degraded agent task outcomes, and the sample (public repos ≥10 stars) excludes the high-complement orgs where the adoption curve is steepest.
  • SourceDoes systematization cause deeper delegation or merely correlate with already-intensive users? The paper shows the association, not the direction. (Not addressed by skill registry to repository lifecycle — it mines artefacts and their diffs, never observing the user or the delegated task, so it can say nothing about direction.)
  • SourceDoes a stale skill measurably degrade agent task outcomes, or do models increasingly route around outdated instructions (Harness Shrinkage as Models Improve)? Gao et al. establish that staleness is widespread and argue stale guidance is "executed rather than read", but never measure the downstream effect; SkillsBench-style evaluation could settle it.
• AI as Primary Author2 open
  • NowThe 60% figure aggregates very different tools and modes (autocomplete acceptance vs. agent-applied diffs). What does "acceptance" mean when the agent applies the change directly and the human's "acceptance" is not reverting it? Sharpened by Review as the Control Point: on GitHub, agent PRs are most often examined only by the developer who invoked the agent (author-only review 40.1% vs 21.5% for human PRs). Whether that counts as review at all is a definitional choice (agent-as-author ⇒ a second set of eyes; agent-as-tool ⇒ self-review) that literally flips the sign of the trend — so "acceptance" and "review" blur into the same unresolved construct. Partially answered: Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? — the construct resolves into a three-way partition by who acts and who looks: affirmative adoption (human applies a suggestion), reviewed non-reversion (agent applies, independent human examines), and bare non-reversion (agent applies, nobody looks — the only rubber-stamp class). The 60% blends all three, which is why it can't answer whether oversight is real; the proposed metric is the partition itself, with only bare non-reversion read as oversight erosion. Still unmeasured: the partition's actual shares in any dataset. Partially answered again, from below (2026-08-12): DECODE measures the affirmative adoption class — the one the partition treats as unambiguous — and finds it is not an endpoint. Of trajectories that begin with a developer accepting a completion, 31% contain a removal edit, retention is bimodal, and the median completion has lost roughly a third of itself within the hour. That does not supply the partition's shares, and it does not touch the two non-reversion classes at all (its unit is a suggestion a human applied, not a diff an agent applied). What it settles is narrower and useful: acceptance is a point on a trajectory, so any partition of it needs a time horizon attached, and the instrument that can see the trajectory is pre-commit editor telemetry rather than anything at PR level.
  • WaitIf agentic authoring crosses from <1% toward double digits, does the whiplash become unmanageable before context-engine tooling matures — or does the tooling mature because of the pressure?
• Building Is Cheap, Arguing Is Expensive2 open
  • SourceWhen does "generate three and compare" become wasteful — at what decision weight is a real argument (or a design doc) still cheaper than three implementations?
  • NowIf design discussion lives in PRs/prototypes, where is the rationale recorded for future readers — does the "why we chose this" knowledge survive, or does it share the staleness problem of Code as Source of Truth? Partially answered: Where Does the Why Live? — it does not survive: discussion buried in merged PRs is orphaned for future readers, and no candidate home holds it durably. What remains open is whether any of the partial patches (richer plan artifacts, a compiled knowledge base) works in practice at team scale.
• Code as Source of Truth2 open
  • NowWhat knowledge genuinely can't live in the codebase (org strategy, the "why," cross-team context) and therefore still needs a durable doc — and how do you keep that small slice current? Partially answered: Where Does the Why Live? — the "why" is the clearest such slice, and every candidate home fails or only partly works; a compiled knowledge base outside the staling code is the least-bad option. The "keep it current" half is untouched.
  • SourceIf onboarding is "ask Claude," what happens to the tacit knowledge that was previously transferred socially in deep-dives — is it captured anywhere, or quietly lost?
• Compute Allocator2 open
  • SourceIs 1% a Thariq-specific number or a regime? For larger, more code-heavy projects the production residue is presumably higher; what sets the ratio?
  • SourceAllocation quality is hard to measure — what's the feedback loop that tells an allocator they spent compute badly (vs. just spending a lot)?
  • ResolvedDoes treating humans as "compute allocators" risk the oversight-fatigue / accountability failure modes the HBR research flags, where the human nominally decides but actually rubber-stamps? Answered: Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? — yes, it is the role's central failure mode, documented at three evidence layers (brain-fry error rates +11%/+39% and the under-engagement swap; Faros's 31.3% no-review telemetry; the CMU theory's P1–P4 load-and-plausibility mechanisms), with two allocator-specific aggravators: allocation quality has no feedback loop, and the human's retained 70% planning share is exactly where rubber-stamping is transcript-invisible. The framing survives only with structural countermeasures — understanding-gated merges (quiz gate), sample-based depth concentrated on high-stakes points, risk-tiered gating — that make understanding rather than signature the merge condition.
• Configurable Human Participation4 open
  • SourceThe "human" is an LLM simulator (GPT-4.1) that also judges — how much of the configuration-dependent structure is a property of human-agent collaboration versus an artifact of GPT-4.1 modeling both sides? The Appendix C.6 simulator-swap is the only check, on a subset.
  • SourceA4 is instantiated as a fixed one-shot proactive intervention. Real proactive humans time their input adaptively — does the "premature/distracting intervention breaks tasks" result hold, worsen, or vanish under a human who chooses when to interject?
  • SourceThe optimal channel is shown to be pattern-specific, but pattern is an oracle label assigned at construction. Can an agent infer which pattern it is in (and thus which channel to solicit) at runtime — the actual deployed skill the paper says agents lack?
  • WaitDoes the "more channels adds coordination overhead" penalty shrink as the backbone improves (a capability gap), or is it a structural cost of mixed-initiative interaction that persists?
• Design by Selection3 open
  • WaitIs "make the last mile manual" durable or transient? Parrott gives two reasons (tokens, and eyeballing beats describing); the token argument dies with cheaper inference, the bandwidth argument shouldn't. Which one is actually load-bearing is testable by watching whether direct-manipulation use falls as models improve.
  • SourceIs the default-aesthetic collapse fixable by context (brand files, moodboards) or is it the novelty ceiling from Why AI Lags at Design reason 3 — i.e. does a fully specified design system still produce house-style output underneath the palette? Partially answered: astryx agent ready design system test (n=1, single practitioner) says fixable where the design system has a slot for the decision, untouched where it doesn't — the model read the brand correctly throughout and only regressed on components with no customization surface. That reframes the residue as a coverage gap rather than a novelty ceiling, but it tests one vendored system, not the underlying question of whether house style persists on fully covered surfaces.
  • SourceTen-options-then-remix assumes the human reliably recognizes the good one. Where does selection break down — does discrimination degrade when all ten candidates are competent, and is there a candidate count past which review cost exceeds authoring cost? Partially answered (adjacent domain, single-candidate case): Post-Acceptance Edit Behavior (empirical, 53.6K in-IDE edits from 1,141 developers) is the first dataset measuring what humans actually do to AI output they already chose. It says selection is revisable and frequently revised — retention is bimodal (kept essentially whole or discarded, rarely in between), 31% of trajectories contain a removal edit, and the sequence identifies when recognition fails: the customize-first path leads to deletion roughly twice as often as the change-functionality-first path. What it cannot answer is this question's actual ask. It observes code completions accepted one at a time, not a ten-candidate set, so it measures neither discrimination-under-uniform-competence nor the review-cost-versus-authoring-cost crossover — and retention records what the developer did, never whether they were right, so nothing in it grades the discrimination itself.
• Design Concept Grilling2 open
  • WaitCan grilling be run AFK against another agent that holds the user's preferences? Pocock's answer in 2026 is "no, this part has to be human-in-the-loop" — but the question is open as agents get better at modeling their principal.
  • SourceHow does grilling change for team work where multiple humans need to align? Pocock's hint: pair-program with the agent in the room, treat it as a third interlocutor.
• Disposable Micro-Apps2 open
  • SourceWhere's the line between a disposable micro-app and tool sprawl? If every edit spawns a bespoke UI, does the workflow fragment?
  • SourceDoes the copy-back-to-markdown round-trip generalize beyond config-shaped data (rules, tables) to richer artifacts?
  • ResolvedCould these micro-apps be templated/reused rather than regenerated — and at what point does that defeat the "disposable" framing and turn into durable tooling? Answered: The HTML Artifact Lifecycle: Where Plan History Lives, and When Disposable Becomes Durable — the correct reuse unit is the generator, not the artifact: a recurring micro-app pattern becomes a skill that regenerates a fresh, fitted app each time, keeping disposable's per-task fit while gaining reuse's consistency (the measured systematization move — skills at 5.4%→26.6% of weekly-active users). An artifact itself graduates to durable only under recurrence + sync-pressure + audience (the design_system.html profile), at which point it inherits maintenance cost, sync cadence, and a seat under the artifact-sprawl bloat ceiling. The failure mode is the un-chosen middle — apps kept around unmaintained: sprawl plus rot with neither fit nor consistency. The discipline is binary: regenerate it, or maintain it; never merely keep it.
• Efficiency Debt of AI-Generated Code3 open
  • SourceDoes the imperative-bias and library-avoidance pattern generalize beyond C++? The paper's own closing question names Rust, Go and Java, and the answer decides whether this is a property of generated code or of C++'s particular idiom space (where the "right" call is a specific absl/std API a generalist model has weak priors on).
  • SourceIs the sub-parity revert rate a property of AI-generated code or of the gates around it? Every reliability figure here comes from a review-gated monorepo with mature static analysis and presubmit CI, and the paper's own reading is that the gates catch the fatal errors. The discriminator is the same measurement in an org with weaker presubmit gates — where Faros's incident numbers come from.
  • SourceStage 3's mean R_eff of 0.385 sits below the 0.5 parity mark, so the best-mitigated regeneration is still on average worse than the human original it replaced. Does any feedback regime — the RLEF pipeline the authors propose, or context injection from the monorepo — push mean R_eff above 0.5, or is sub-parity efficiency a floor for generalist models on performance-sensitive code?
• HTML as the New Markdown1 open
  • SourceDoes this generalize past one expert practitioner, or does it require Thariq-level fluency with Claude to be worth the overhead?
  • ResolvedDoes the human-facing harness keep growing without bound, or does it hit its own bloat ceiling (an HTML plan too elaborate to read, like the markdown it replaced)? Answered: Does the Human-Facing Harness (HTML Artifacts) Hit Its Own Bloat Ceiling? — yes; HTML raises and reshapes the human-attention ceiling but can't remove it, and the bloat relocates from document-length to artifact-sprawl/rubber-stamping.
  • ResolvedHTML is heavier to diff and version than markdown — what happens to plan history and review when artifacts are single-file websites? Answered: The HTML Artifact Lifecycle: Where Plan History Lives, and When Disposable Becomes Durable — the artifact is a compiled view, not a record: version the content layer (the copy-back round-trip and the extract-from-code pattern already do this), regenerate the presentation on demand, and reattach review to decisions rather than diffs (the plan-ordered-by-likelihood-of-change technique puts the reviewable delta at the top). Presentation history is deliberately discarded — regenerable at abundance prices, it isn't worth versioning — and a presentation choice that becomes load-bearing has by definition graduated to durable tooling with real versioning obligations. Residual (tracked above): no source yet documents team-scale multi-author HTML-plan review.
• Living Design System3 open
  • SourceHow does the design_system.html stay in sync as the codebase evolves — re-extract on a cadence, or wire it into CI? And is completeness (does every component the product needs have a slot?) the binding constraint rather than freshness — the Astryx test suggests uncovered components fail silently, which staleness checks wouldn't catch.
  • SourceDoes a rendered, model-readable design system measurably improve on-brand output vs. a plain CSS/token file, or is the win mostly human legibility? Partially answered: astryx agent ready design system test shows structured artifacts (DESIGN.md + layered tokens) flip output from generic to on-brand, so the artifact clearly beats no artifact — but it does not isolate the variable this question actually asks about, since the artifacts used were plain files, not a rendered page. Rendered-vs-plain remains untested.
  • SourceAt what project size does maintaining the artifact cost more than the consistency it buys?
• LLM-Assisted Grey-Literature Theory Building3 open
  • SourceThe authors couldn't locate the saturation point because synthesis was manual — how few documents actually suffice, and can a cheaper sample match the 3,100-doc theory?
  • SourceAutomating the codes→theory step failed with a naïve bottom-up prompt; is that a prompt/scaffolding limitation or a genuine ceiling on LLM interpretive synthesis over thousands of codes?
  • SourceThe three-lens design manages coder bias, but the relevance judge and segmenter are single-model — do those upstream gates impose their own systematic slant on what reaches the codebook?
• Outsource Your Thinking, Not Your Understanding2 open
  • WaitKarpathy's open frontier: can "understanding" itself eventually be automated, or is it definitionally the human residue? His "back in a couple years" hedge leaves it open.
  • SourceIf understanding is the bottleneck, is the highest-ROI skill learning how to build understanding fast (knowledge-base hygiene, asking the right projections) — and can that be taught?
• Planning / Execution Division of Labor3 open
  • WaitDoes the human share of planning decisions fall over time as models improve (the ceiling rising into the planning layer), or is ~70% a stable human floor?
  • Source"Decision attribution" is inferred from transcripts. When Claude proposes a plan and the user assents, is that scored as the user's planning decision or Claude's? The rubber-stamping boundary is exactly where the measure is hardest. Partially answered on the execution half only (2026-08-12): DECODE shows the inference is avoidable below the planning layer — an edit trajectory records what a developer changed in an accepted completion as a byte-level fact, with no transcript reading, and 56% of those edits change functionality rather than naming. The rubber-stamping boundary this question actually asks about is untouched, because assent to a proposed plan leaves no edit trace at all; the residual claim is that the hardest attribution problem is specific to planning, and that the execution share is measurable without a classifier if you instrument the editor rather than the conversation.
  • SourceHeadless/SDK/pipeline usage (excluded here) is where execution autonomy is highest and planning is front-loaded into a single prompt — does the 70/20 split survive there, or collapse toward full delegation?
• Post-Acceptance Edit Behavior3 open
  • SourceThe completion pool is 2024-to-early-2025 inline autocomplete, median 9 lines. Do the bimodal retention shape, the 15-minute knee and the ~31% removal-edit rate hold at 2026 agentic granularity, where the unit is a multi-file diff the developer never watched being written? The discriminator is the same trajectory extraction run over agent edits rather than completions.
  • SourceThe customize-then-remove path (23.4%, against 12.2% after a functionality edit) is offered as evidence that subtly misaligned completions resist adaptation. A reading-depth explanation predicts the same matrix: customizing requires reading the completion closely, and close reading is when its real flaw surfaces. Discriminating them needs a signal outside the edit stream — time-to-first-edit conditioned on completion length, or an eye-tracking or dwell-time proxy. Which mechanism is right decides whether the fix is better generation or earlier forced inspection.
  • SourceIs completion retention predictable in principle? Fine-tuning lifts classification only to F1 0.45 against a 0.33 random baseline, and generation on the dominant edit class (changing functionality, 56% of snapshots) tops out at 0.49 Levenshtein similarity for every model tried. Either the signal is in context the models were not given (repository, task history, the developer's other files) or retention is a property of intent that no amount of code context contains — and the paper's proposed "detect low-editability generations before showing them" product depends on which.
• Review as the Control Point4 open
  • SourceEvery one of the 67 relationships is a hypothesis, not a finding — the paper's explicit call is for causal-estimand studies (controlling for the other constructs) to confirm, reverse, or drop each edge. Which of P1–P17 survive measurement? Partially answered (direction only, no edge confirmed): Security Debt of Agent-Generated Code (empirical, arXiv 2607.12428) supplies outcome-level evidence pointing the way P1 (load → shallower review) and P4 (surface plausibility disarms the reviewer) predict — 81.1% of genuine credentials in agentic PRs drew no reviewer comment before integration, and the reviewer-focus finding it cites (Haider & Zimmermann, arXiv 2601.19287) is that inline comments on AI-authored code address logical and functional correctness rather than security posture. But it isolates no mechanism, controls for none of the other constructs, and has no human-PR baseline, so it corroborates a direction without confirming an edge. First production test, and it is a null (2026-08-12): Tran et al. have the human baseline the security paper lacks and find no correlation between review time or iteration count and the survival of inefficient AI-generated code — so on the one outcome class they measured, the P1 chain does not reach the outcome. It is reported without a statistic or specification, and it tests one defect class in one review-mature monorepo, so it does not reverse P1 either. The usable result is narrower and new: the mechanism map needs a defect-class dimension, because review depth cannot plausibly moderate what review cannot see. The dimension gets its positive pole the same day: Dipongkor et al. (empirical, 4,882 agentic PRs) identify a class where attention does have purchase and say where to point it — error-handling constructs, unexercised 81.0–86.0% of the time in both languages whether or not the agent wrote tests. So the two ends of the dimension are now instantiated rather than merely postulated: a class review cannot catch at any depth, and a class review can catch if told where to look. P8/P9 remain untested (2026-08-12): Cynthia et al. is the largest study yet of automated reviewers but measures the adoption of their comments (71.4% pooled), not the throughput P8 claims or the quality P9 contests — so the two edges nearest to automated review still have no measurement, only a new third quantity beside them. A fourth quantity, and the closest approach yet to P9 (2026-08-12): Greptile measures automated reviewers' recall on ~1,500 labelled high-severity bugs (52–62% by arm) — not adoption, not throughput, and nearer the quality half of P9 than anything before it, but still not a quality outcome, since it scores detection against a self-built label set rather than defects that shipped. It is case-study, not empirical, and it moves the moderator rather than the edge: automated-reviewer capability turns out to depend on which model authored the code under review.
  • WaitDoes the no-review convergence hold as agentic authoring crosses from <1% of PRs toward double digits, or does the early-adoption discipline break down under volume the way Faros predicts?
  • SourceThe paper's own question: which decisions, under which conditions, push the system toward the virtuous loop rather than the vicious one? — the system-dynamics leverage-point analysis it gestures at but doesn't run.
  • SourceThe three contested edges (automated review → quality/security, P9; governance → latency, P17; and one more) are contested because their sign is moderator-set — what are the moderator thresholds that flip them?
• Risk-Tiered Auto-Approval3 open
  • SourceThe case study reports volume and never efficacy. What is the escaped-defect or incident rate of auto-approved PRs versus the human-stamp baseline the Slack channel used to produce? PostHog has both populations in its own history, which makes this a checkable before/after rather than a request for new instrumentation. The same history answers a second question for free (added 2026-08-12): plotting the share of merged PRs that would clear the 500-line/20-file ceiling, quarter by quarter, dates how fast the gate's coverage decays as the ambient PR-size distribution rises.
  • SourceA keyword deny-list is a proxy for blast radius, and Security Debt of Agent-Generated Code shows the proxy misses where the measured debt concentrates (CI/container plumbing, 87.6%). Does adding CI/IaC paths to the deny-list restore coverage, or does it shrink the auto-approvable set so far that the gate stops paying for itself? A third candidate predicate is now priced but not tested (2026-08-12): Dipongkor et al. (empirical) show that "the existing suite covers this diff" is computable, deterministic, and rarely true — 27.0% of changed lines in Python, nothing at all in 64.8% of PRs. That reframes this bullet as a choice between two extensions with the same failure mode rather than one open question: both a path deny-list and a coverage floor buy precision by shrinking the auto-approvable set, and neither has been measured against the volume it costs. The same before/after PostHog already has in its history would answer both at once.
  • SourceIf the size ceiling becomes a design target — agents instructed to emit stacks under 400 lines — does total risk fall, or does it just redistribute into more PRs each below the gate, with the integration risk moving to the seams between them? The security study's size gradient is measured per-PR and cannot distinguish these.
• Same-Model Review Blindness3 open
  • SourceEvery number here rests on a ground truth Greptile built from "sentiment analysis, upvote/downvote ratios, and git archaeology," with no protocol, no agreement statistic, no released artifact, and an LLM judge doing the matching without published validation. Does the crossover survive on a label set someone else constructed — a curated defect corpus with human adjudication, or an injected-bug benchmark where ground truth is golden by construction? Until then the direction is a vendor's finding and the magnitudes are uncheckable.
  • SourceIs the blindness a property of shared training lineage or of stylistic fit? The result is compatible with a much duller explanation: each reviewer happens to be strong on the bug mix the other agent produces, for reasons unrelated to authorship. The published category data does not settle it (composition reproduces only ~7% of the effect, which argues against the dull reading but is not a test of it). The discriminating experiment is cheap and Greptile has the datasets: run a third-family model — Gemini, or an open-weight reviewer — across both corpora. If lineage is the mechanism, the third model has no same-model arm and should score near the cross-model band on both.
  • WaitCaridad predicts the effect shrinks as models converge: "a year ago, the performance difference in the opening figure would likely have been larger." Does the same protocol, rerun on the next generation of both families, show a narrower gap? The trigger event is the next paired frontier release measured the same way; note that the prediction is also the one Greptile's own product would least like to be true.
• Security Debt of Agent-Generated Code3 open
  • SourceThe 38.9% smell rate has no human-authored-PR control over the same high-risk paths, and the 2.7×-vulnerability figure it leans on traces to a vendor blog and a Substack post. Does agentic authorship raise smell density, or merely raise the volume of CI/IaC files touched? A matched human baseline on the same path set would settle it. Partially answered 2026-08-12 by Tran et al. (empirical, Google, arXiv 2608.06640): the corpus now has a matched human baseline — 3.52M production changes with authoring-time provenance and a human-written cohort — and on its taxonomy AI-generated code is below parity on Correctness and Safety (0.94x), API misuse (0.93x) and lifetime/ownership hazards (0.71x), with the excess concentrated in efficiency and interface coupling instead. That is direct evidence against the 2.7x vulnerability figure this page marks as uncorroborated. It is not the same path set: application C++ scored by clang-tidy categories, not GitHub Actions / Dockerfiles / IaC scored for security smells, and the AI cohort there does touch more files per change (3 vs 2 median), which is the volume half of this bullet left unmeasured. So the density-vs-volume question survives, with the prior it was testing now leaning the other way.
  • SourceDoes "no reviewer comment" mean undetected? 60 of the 74 genuine credentials were removed without a comment, yet the abstract reads the same rows as detection failure. Commit-history analysis of who removed them and when is a checkable discriminator between silent remediation and coincidental churn.
  • SourceReview coverage of agent PRs is converging toward the human baseline while efficacy on credentials sits near zero. Do the two trends move together as teams mature, or independently — i.e. does showing up to review buy any measurable catch-rate improvement? Not answered by the closest-looking evidence (2026-08-12), and the resemblance is a trap. Cynthia et al. (empirical, arXiv 2607.21997) is the largest efficacy-adjacent measurement of the review layer to date — 54,713 agent review comments, 71.4% resolved — but it measures the opposite direction: the agent reviewing a pull request and the human deciding whether to act, rather than a reviewer catching defects in agent-authored code. Its resolution rate is an adoption proxy, and its own construct-validity section concedes that "comments may be resolved without being useful or remain unresolved despite being valuable." What it does supply is the negative half of the diagnosis: developers in its argued sample are engaged, not asleep, so whatever suppresses catch-rate on this page's credentials is unlikely to be plain inattention. The question still needs a study that pairs review presence with a defect ground truth on the same PRs.
• Telemetry vs. Survey Measurement2 open
  • Is there a non-vendor telemetry dataset large enough to adjudicate the maturity-protection question independently of Faros's commercial framing? Partially answered: CMU's arXiv 2607.07980 supplies exactly this — a non-vendor, 2.5M+-PR GitHub telemetry study — and it (a) finds the agent no-review rate converging toward the human baseline rather than a widening gap, and (b) argues the effect's sign is set by team practice, closer to DORA. The catch: its own headline is that the telemetry is direction-unstable, so it counterweights Faros without cleanly settling the maturity question — the honest verdict is "surface telemetry alone, vendor or not, can't adjudicate this." The worked example of that verdict is The Under-Review Divergence: Faros's Widening Crisis vs. CMU's Convergence: the two telemetry studies' opposite under-review headlines dissolve once metric, population, time axis, and authorship unit are aligned — the datasets never disagreed, only the framings did.
  • SourceDoes anchoring an adoption survey's definition of "AI" change the answer, and in the predicted direction? Kalff & Simbeck document respondents excluding predictive ML from the category while counting ChatGPT in it, which would bias reported adoption down for exactly the tool class their headline finding is about. Falsifiable two ways: a split-ballot survey where one arm names the underlying technique ("software that scores turnover risk") instead of the label, or validating firm-level self-reported adoption against vendor-license/spend records for the same firms — the Ramp instrument applied as a criterion rather than a substitute. Related evidence (2026-08-12), not an answer: DX reports a self-reported AI-generated code share of 34% to 52% across Q1-Q2 2026 in 500+ organizations, while Google's authoring-time provenance over the overlapping window reads 28.99% to 68.62%. Different populations (a cross-industry customer panel vs one C++ monorepo), different constructs (code share vs adoption), no matched firms, and DX's question wording sits in a gated PDF the vault does not hold — so this settles nothing. It is nonetheless the corpus's first side-by-side of a self-reported code share against a provenance-measured one, and the direction is the one Kalff & Simbeck's construct-collapse mechanism predicts: self-report reads below the instrument that counts bytes, not above it. The criterion validation this bullet asks for is that same comparison run on the same firms.
  • ResolvedSurveys and telemetry measure different things (felt productivity vs. system outcomes); is the "contradiction" partly a category error — both true at their own layer — rather than one being wrong? Answered: Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? — yes for the measurement halves: surveys capture felt productivity (genuinely real at the individual layer — task completion did rise) while telemetry captures system outcomes that haven't propagated into feeling yet; during a fast transition the layers legitimately diverge, and the AEI's linked telemetry+survey design already treats them as complements. But not entirely: the maturity-protection claim is one proposition about one layer (system outcomes) and remains substantively contested — Faros's "no protection" aligns with its vendor incentive, CMU's moderator theory argues the sign is team-set (closer to DORA) but measures no maturity effect. The category-error dissolution cleans the framing; the one real disagreement stays open (tracked in the non-vendor-dataset question above).
• The Three Loops of AI-Native Building3 open
  • NowNg asserts the developer's QA burden fell "significantly." Faros's 2026 telemetry measures the opposite for production orgs. Is the split really 0-to-1-vs-production, or is Ng's self-report subject to the same optimism bias the survey literature keeps finding? Partially answered: Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? — both-and, not either/or. The scope split is real and does most of the work (0-to-1 builds lack the review functions that make verification expensive in production: no queue, no incident budget, no future maintainer needing comprehension), so Ng's burden could genuinely fall; simultaneously his evidence is self-reported felt burden, the instrument class shown to lag system reality and skew rosy (The Automation–Optimism Link's no-deficit self-reports vs the measured vanished gains in Contractor & Reyes's randomized study), so "significantly" is a feeling, not a magnitude. The Faros-outranks-both tiebreak for the org case stands. Missing: any measured QA-time series for 0-to-1 builders. The consequence half gets a number, 2026-08-12 — DX's Q2 2026 panel (vendor-claim, 500+ organizations) reports AI users saving an estimated 4-6 hours per week while the innovation ratio (share of time on new features versus maintenance and overhead) stays flat. That is a partial concession to Ng and a rebuttal of what he draws from it: the hours really do come free, and at panel scale they are not landing where his account says they go — on higher-level product decisions. Note what it is not. It is a vendor's self-selected customer panel with the methodology in a gated report, it measures time allocation rather than the QA burden itself, and a flat ratio is consistent with the freed hours being consumed by the review and incident load Faros measures rather than with them never existing. The 0-to-1 scope split survives it untouched, since a solo builder has no innovation ratio to move.
  • SourceThe external loop is the unshortened one. Is that physics (users take time to react) or an unautomated frontier (synthetic users, deployment simulation applied to products rather than models)?
  • WaitIf the human's presence in the middle loop is justified by a context advantage that is closable, the middle loop is a transitional structure. What does a two-loop world look like — and who translates the external loop's signal then?
• The Verifiability Thesis2 open
  • SourceWhere's the boundary of "council of LLM judges" reliability — does it hold for genuinely contested value judgments, or only for quality/coherence? Partially answered (2026-08-04) by Zhou (2026), and it inverts the question's premise. The question assumes the council is safe on the easy end and asks how far up it holds; the measurement says it fails on the easiest end — objective correctness on grade-school math — once anything optimizes against it. Three cross-family judges accepting only unanimously still pass 55% of manufactured wrong answers, and Proposition 2 shows no monotone rule over a shared plausibility signal can do better. Two further findings sharpen where the boundary actually sits. The council is fine as a static rater and fails as a reward: the same judges hold usable discrimination (0.21–0.38) before optimization and collapse to 0.05–0.17 after, so the binding variable is optimization pressure, not the contestedness of the judgment. And reference-free verdicts track prompt framing rather than correctness — with unit-test ground truth held fixed, Llama's gap@16 swings −0.106 under a strict instruction to +0.722 under a lenient one, so on the fuzzy end there may be no stable operating point to have a boundary about. And the council's headroom is small before any of that. Yang et al. (2026) measure juror error correlation on ordinary preference grading with nothing optimizing against the judges — ρ = 0.944–0.972 for repeated samples of one judge, 0.664–0.706 across a stronger family, and family-mixed juries also below independence predictions — so five jurors buy 0.463 → 0.482 on LLMBar. Condorcet's amplification requires independent voters and LLM judges are not that, optimization or no. The council was never carrying the weight the thesis assigns it; optimization pressure only makes the shortfall adversarial. What is not answered: nothing here tests contested value judgments, where there is no anchor to audit against and hence no way to run this measurement at all.
  • SourceThe "labs care" dependency is fragile: capabilities can appear or stagnate based on lab priorities you don't control. How should a product hedge against the data-distribution rug-pull?
• Unknowns as the Agentic Bottleneck4 open
  • SourceIs "the first model bottlenecked by my unknowns" a property of Fable or of Thariq? A frontier-lab engineer with deep model fluency hits the human-side ceiling before an average user does — which would make this a leading indicator rather than a current universal.
  • SourceThe quiz gate is self-administered and self-graded (by the model, on the model's own work). What stops a comfortable equilibrium where the quiz gets easier as the reviewer gets lazier? Cf. the maker/checker problem in Verification as the New Bottleneck. (The drift the question worries about is now measured in one register — Security Debt of Agent-Generated Code finds humans committing 67.6% of the genuine leaked credentials inside agent PRs, read by its authors as reduced vigilance. That establishes the direction is real; it says nothing about what would stop it, so the question stands unanswered.) Partially answered from an unexpected direction (2026-08-12), and it moves the defect earlier: Greptile (case-study) measures a model's recall on high-severity bugs in code its own family authored at 6–12 points below its recall on the other family's code. The quiz gate does not need to decay to be weak — a self-authored quiz asks about what the authoring model thinks matters, and the categories it under-weights in review are correlated with the ones it under-weights in authoring, so the blind spots are missing from the question set on day one. That is a different failure from the equilibrium this bullet asks about, and it comes with the obvious mitigation attached (a different model family writes the quiz). The equilibrium question itself — what stops the quiz getting easier as the reviewer gets lazier — is still unanswered; nothing measures a self-graded gate over time.
  • SourceElicitation has a cost. Every technique here spends a session's worth of tokens and attention on not building. Nothing in the source bounds when the blindspot pass costs more than the bug it prevents.
  • SourceIf unknown knowns are extractable, are they extractable once? Does a codified blindspot pass become a skill file that permanently narrows the gap, or does each new territory reopen it?
• Verification as the New Bottleneck2 open
  • NowFung's own open question: "How far do you push fully automated reviews?" — where's the speed/safety balance, and how do you keep humans confident without re-introducing the review bottleneck? Sharpened by Review as the Control Point: full automation reliably raises review throughput and cuts latency (its P8), but its effect on code quality and security is contested (P9), and the latency effect of a review-governance policy flips sign by calibration — a risk-tiered policy that gates only material changes lowers latency, a blanket policy raises it (P17). So "how far" has no single answer: the safe frontier is set by automated-reviewer capability and process design (two of that page's three moderators), not by a fixed dial. Partially answered: Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? — "how far" is a partition, not a dial: automate 100% of mechanical checking (style, lint, spec-drift, tests); the contested zone is automated quality/security judgment (P9, vendor claims unmeasured); and the hard limit is not defect-catching but the functions review performs besides it — reviewer-skill growth (P12/P13), collective ownership (P14), comprehension-debt paydown (P15) — which erode under automation even if the machine catches every bug. Residual: the moderator thresholds are still unmeasured. Floor datum added 2026-07-29 by Security Debt of Agent-Generated Code (empirical): on hard-coded credentials — the one smell class with purpose-built automated detection — seven distinct bots and human reviewers together commented on just 18.9% of the genuine live credentials in 4,022 agentic PRs. So the "automate mechanical checking fully" half of the partition is a prescription, not a description: where it is fully mechanizable, it currently isn't mechanized well. Deployed calibration added 2026-07-29 by Risk-Tiered Auto-Approval (case-study): PostHog's StampHog answers "how far" as as far as the cheap structural checks reach — PR state, a blast-radius deny-list, and a <500-line/<20-file ceiling gate the decision, with an LLM check demoted to a last-position veto that may tighten but never loosen — and that reaches ~1 in 3 PRs merged into their main repo (1.6K in a month). Two qualifiers keep this from being an answer: what it replaced was a Slack stamp-exchange ritual by engineers with "little to no context," so it converts an implicit rubber stamp into an explicit gate-checked one rather than automating substantive review; and the account reports volume with no false-approval or escaped-defect rate, which is the contested half of P9 left unmeasured at scale. A different kind of answer, 2026-08-12 — Tran et al. (empirical, 3.52M production changes with a human control cohort): for at least one defect class, "how far" is the wrong axis, because the human pass contributes nothing measurable to catching it at any depth. They tested review time and iteration count against the survival of inefficient AI-generated C++ and found no correlation, concluding that upstream automated intervention is necessary rather than merely faster. That reframes the partition above: it is not only "mechanical checking vs judgment," it is also which defects are legible to a reader at all — a missing move constructor inside a correct, idiomatic function is invisible to attention and trivially visible to a static category. The caveats are that the null is reported without a statistic or specification, and that a monorepo with mature static analysis has already mechanized much of what review would otherwise catch. The adoption number arrives, 2026-08-12 — Cynthia et al. (empirical, 54,713 agent review comments across 341 repos): where the partition above says "automate mechanical checking fully," this is the first population-scale reading of whether the automated layer's output is taken. It is, roughly seven times in ten (72.9% Copilot / 67.2% Cursor / 54.8% Codex), and the lever that moves it is actionability, not eloquence — an inline code suggestion is the strongest predictor (OR 1.62) while length and sheer explanation count hurt. Two things keep this from extending the partition further. The pooled model's AUC is 0.58, so comment design explains very little of the outcome and the agent-level spread survives controlling for it. And adoption is not efficacy: the study measures whether a human closed the thread, never whether a defect existed — so it fills in "does the machine's output get used" and leaves "does it catch anything" exactly where the 18.9% credential floor left it.
  • SourceIf CI/build is the hidden jam, does verification infrastructure (test runners, CI capacity) become the actual capex of an AI-native org? Partially answered 2026-07-29 by Agent-Generated Test Quality (empirical): it supplies the mechanism but not the cost. Agent-authored tests in AIDev carry flakiness indicators — unmocked file I/O, random, datetime.now — at a 0.44 rate vs 0.30 for human-authored tests, so the throughput increase arrives with a compounding rerun tax on the runner rather than a one-off load increase. Two gaps keep this short of an answer: the study measures candidate rate (its specified dynamic re-run stage is never reported), and nothing in it prices CI capacity, so the jam is evidenced while the capex claim is not. A CI-spend-per-merged-PR series stratified by agent authorship is what would settle it. Priced, but only by a vendor, 2026-07-29 by circleci q2 pulse 2026 (vendor-claim): CircleCI now supplies the cost half the study omitted — a countable cycles-to-merge metric (median MER 3.9 vs 1.3 for its elite cohort) and a modeled ~$900K/yr delivery cost for a 50-developer team, ~$700K of it claimed recoverable by shifting checks into the inner loop, including a "token reload penalty" for agents idling on CI. That is the first attempt in the vault to put a currency figure on the jam, and it is an argument that verification infrastructure is a real capex line. It does not settle the question: the $900K is a model over CircleCI's own customers with no published inputs, the recoverable figure is the sales case for CircleCI's inner-loop products, and it prices CI time and tokens rather than the runner-capacity build-out the question asks about. The stratified non-vendor spend series is still the thing that would settle it.
• Vertical Slice Tracer Bullets1 open
  • SourceHow should slice granularity be tuned? Too thin = many merge conflicts; too thick = back to horizontal.
  • ResolvedCan the planner agent be trusted to slice vertically once told to, or does it need a verifier that flags horizontal slices? Pocock's experience: it needs the verifier, at least through 4.7. Answered: Verifying Without a Compiler: Cowork's Harness vs Claude Code's, and Why the Slice Verifier Stays — it needs the verifier by design, not just empirically: slice shape is a mechanically checkable invariant (does the ticket touch schema + service + UI?), and checkable invariants belong in the deterministic checker regardless of model trust. "Slice vertically" in a prompt is a behavior request — unreliable against the training prior on the way up, compounding-prone once the behavior goes native — while the verifier is a constraint: it doesn't compound, costs ~nothing, and catches drift in either direction. Correct trajectory as models improve: prune the prompt line when ablation shows it's native; keep the checker, the way tests outlive the model learning to write correct code.
• Vibe Coding vs. Agentic Engineering2 open
  • SourceKarpathy hints at "one domain that's very [valuable]" for founders but won't say which (didn't want to "vague-post on stage"). What verifiable RL-environment domain is he gesturing at?
  • WaitIf the mediocre/AI-native spread keeps widening, what does that do to team composition — a few extreme outliers plus agents, vs. broad mid-level staffing?