Risk-Tiered Auto-Approval

Sources#

• Stop being the code review bottleneck
• The State of AI Impact in Engineering: Q2 2026

Summary#

Risk-tiered auto-approval is a merge gate that decides whether a human needs to look at this PR at all, using cheap structural properties of the diff rather than a judgment about its correctness. PostHog's implementation — StampHog, described by Jina Yoon in the PostHog newsletter, July 2026 — is the vault's first documented production instance at scale: it gave the final approval on roughly 1 in 3 PRs merged into PostHog's main repo over a quarter, and handled 1.6K PRs in one month.

It is the deployed form of the construct CMU's review theory names in P17: a risk-tiered governance policy that gates only material changes lowers review latency, where a blanket policy that gates everything raises it. That page argues the moderator exists; this source shows one calibration of it running against a real merge queue.

Evidence note. case-study — a first-party account of the team's own tooling, written for a newsletter that promotes the practice. The volume figures (1 in 3, 1.6K/month) are the kind of thing a team can count reliably and there is no obvious incentive to inflate them. What the account never reports is the number that matters for safety: no false-approval rate, no escaped-defect count, no incident attribution. Treat the mechanism as documented and the outcome as unmeasured.

The gate stack: deterministic first, model last#

An engineer opts a PR in by adding a stamphog label. Four checks run:

1. PR state — no merge conflicts, no outstanding changes-requested review.
2. Blast radius — a deny-list of keywords (auth, secrets, billing, public APIs, …). Any hit routes to a human.
3. Diff size — under 500 lines and 20 files.
4. A simple LLM check — "for basic showstoppers."

The ordering is the design. Three of the four gates are deterministic properties of the diff and the repo; the model runs last, on a change that has already been established to be small and to touch nothing on the sensitive list. This inverts the default shape of AI code review, where a model is the detector and its judgment is what stands between the diff and the merge — the arrangement whose quality effect P9 marks as genuinely contested. Here the model is a veto on an already-narrowed set, not the thing doing the narrowing. It is the same structural move "prevent, don't detect" makes in the security register, arriving independently from the throughput side.

Output is deliberately minimal on both branches: approval is a bare GitHub approval with no line comments (it is explicitly not pretending to be a review), and refusal is a 1–2-sentence reason, a risk-level rating, and next steps.

Three invariants#

PostHog's own porting prompt states the safety contract it wants preserved verbatim: "fail closed, never request changes or merge, LLM can tighten gates but never loosen."

• Fail closed — ambiguity resolves toward the human, so the failure mode of a broken gate is added latency rather than an unreviewed merge.
• Never merge, never request changes — the agent's authority is bounded to one action, the approval. It cannot land code and it cannot block a human's PR with a changes-requested review. Compare Claude Code Auto Mode, where the classifier's authority is likewise one-directional (it may block a tool call, never unblock a prohibited one).
• The model may tighten but never loosen — the asymmetry that keeps a persuadable component from becoming an attack surface. An LLM that can only move a decision toward more human involvement cannot be talked into an approval by anything in the diff, which matters because the diff is attacker-controlled content in the general case (Agentic Prompt Injection).

Escalation is routing, not blocking#

A refusal doesn't dump the PR back into a queue; it routes to a subject-matter expert, selected by "CODEOWNERS-soft and git-blame familiarity." The gate's real product is therefore an attention-allocation decision, not an approval — it partitions the merge stream into "nobody needs to see this" and "this specific person should." That is the oversight-fatigue mitigation ("concentrate review on high-stakes decision points rather than every output") mechanized at the merge boundary, and it is why the throughput gain isn't purely subtractive: attention removed from 1.6K low-context stamps is available for the escalated remainder.

What it displaced matters for how you read it#

Before StampHog, PostHog ran a Slack channel (#dev-stamp-exchange, with a leaderboard) where an engineer dropped a PR and waited for someone to approve it. Yoon's own description of the cost: "each stamp required another engineer to take themselves out of their flow to approve a change they had little to no context on."

This reframes what the automation replaced. It was not substantive review being handed to a machine — it was an approval ritual that already carried near-zero information, performed by the least-informed available human. In the three-way acceptance partition from Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping?, the stamp exchange was already producing something close to bare non-reversion wearing a signature.

The honest reading, then, is that StampHog converts an implicit rubber stamp into an explicit, logged, gate-checked one. That is a real improvement in accounting — the auto-approved set is now enumerable, its criteria are written down, and the deny-list is a reviewable artifact — without being evidence that automated approval substitutes for review that was doing work. Reading "1 in 3 PRs approved by an agent" as "a third of review was automated away" over-claims; a large share of that third was ceremony.

Calibrate locally, or the deny-list is decoration#

PostHog's porting prompt refuses to hand over its thresholds: "Their deny-list and thresholds are calibrated to their codebase — re-derive for me: mine my git history for high-blast-radius deny candidates and calibrate size/tier ceilings from my merged PRs."

This is the load-bearing operational caveat. Both discriminating gates are empirical facts about one repository: which identifiers mark danger, and what "small" means in that codebase's merged-PR distribution. A deny-list copied across repos matches the wrong strings, and a 500-line ceiling imported into a repo whose median PR is 40 lines gates almost nothing. The transferable artifact is the derivation procedure (mine history for blast-radius candidates, fit ceilings to the merged-PR distribution), not the config.

And re-calibrate: the distribution the ceiling sits on is moving#

The porting prompt says to calibrate size ceilings from your merged PRs. It does not say to re-calibrate, and the ambient distribution is not holding still. DX's Q2 2026 panel (vendor-claim, 500+ organizations) reports median PR size nearly doubling between Q1 and Q2 2026, corroborating Faros's +51.3% on a calendar axis rather than an adoption cross-section.

A constant ceiling on a rising distribution loses coverage silently. Nobody edits the config, nothing errors, and the share of the merge stream that clears 500 lines and 20 files just shrinks — so "~1 in 3 PRs" is a measurement of one quarter, not a property of the design. Two consequences, pointing opposite ways:

• The decay is safe. Shrinking coverage under a fail-closed gate produces latency, not unreviewed merges. This is the invariant earning its keep against a threat its authors did not name.
• It makes the stacked-PR practice load-bearing rather than optional. The size gate only keeps paying if the authoring side is actively pushed down the size curve; otherwise the distribution walks out from under the ceiling and the gate quietly retires itself. Decomposition is what holds the auto-approvable band open.

The operational form is to express the ceiling as a percentile of the repo's own recent merged-PR distribution, recomputed, rather than as a constant — which also keeps it aligned with the 16.2%-to-53.6% smell gradient, since that gradient is a fact about absolute diff size and does not move when your median does. Caveat the transfer, though: DX measures a cross-industry panel median, not PostHog's repository, and no source measures whether any specific repo's distribution shifted.

The upstream complement: shrink the diff to fit the gate#

The source's fourth pattern turns the size gate into a design constraint rather than a filter. Daniel Visca's rule is observability over reasoning: "Agents are good at explaining why their code works. The explanation is often convincing... but also wrong" — so don't accept the argument when you can watch the behavior. Observation doesn't scale to a 3,000-line PR, so the agent is instructed to decompose the work into a stack of small single-purpose PRs (target <400 changed lines, stacked via Graphite), each independently runnable, each shipping "a command to run and the output I should expect," merged bottom-up so every layer builds only on already-observed behavior.

The two patterns compose, and the source says so: decomposition "lets StampHog from #3 auto-approve the small and focused PRs." The result is two checks of different kinds — the agent reasoning about the code, and a human observing it run — rather than two passes of the same kind. This is the merge-queue instance of the agent's explanation being the weakest available evidence, and the practical answer to the size gate's obvious objection ("most real work exceeds 500 lines"): change the unit of work, don't raise the ceiling.

Tension with the measured detection floor#

Security Debt of Agent-Generated Code (Sakib et al., arXiv 2607.12428, empirical) measured the surface this design gates on, and it cuts both ways. The empirical source outranks this case study wherever they meet — one is a coded study of 4,022 PRs, the other a first-party account with no efficacy measurement — so both readings below are the paper's, applied to PostHog's design.

It supports the size gate, strongly. Security-smell prevalence in agentic PRs climbs monotonically with change size, 16.2% for 1–9-line PRs to 53.6% for 1000+-line PRs. A 500-line/20-file ceiling keeps auto-approval inside the low-prevalence band, and the stacked-PR practice deliberately pushes work down that curve. That is independent, non-vendor, empirical backing for the one gate in the stack that looks most arbitrary — and it is the size-stratified evidence Faros asked for and could not produce.

It undercuts the deny-list, sharply. The deny-list is organized around business-risk topics (auth, secrets, billing, public APIs). The measured smell mass is somewhere else entirely: 82.3% is supply-chain integrity — mutable action tags, unpinned installs — and 87.6% of all smells live in GitHub Actions workflows and Dockerfiles, files that read as boilerplate and match no business-risk keyword. An unpinned actions/checkout@main in a 30-line workflow diff clears PR state, clears the deny-list, clears the size gate, and arrives at the LLM showstopper check as the only remaining defense — the layer whose measured performance on the most mechanically detectable smell class (hard-coded credentials, seven distinct commercial detectors present) was a comment on 18.9% of genuine live credentials. Build-and-deploy plumbing is precisely the blind spot of a deny-list built from application-risk intuitions, and it is where the debt actually accrues.

The concrete prescription this yields: .github/workflows/, Dockerfile, and IaC paths belong on the deny-list as paths, independent of keywords. PostHog's own instruction to mine git history for deny candidates would likely surface them; a copied deny-list of business-risk words never will.

The rival predicate: "the tests cover this change"#

The deny-list is a proxy for blast radius, and the obvious third proxy — the existing suite already exercises this diff — is the one gate in this space that is a fact about the change rather than a guess about it. Dipongkor et al. (arXiv 2607.18057, empirical) price it on 4,882 agentic PRs, and the price is low: existing tests execute 61.5% of agents' changed executable lines in Java and 27.0% in Python, with 64.8% of Python PRs having no changed line executed by any existing test at all. Their own prescription to practitioners is the negative form of it — "teams using agentic PRs should not assume a passing run of tests means that the change has been tested."

Two consequences for this gate stack, pointing opposite ways.

Against the stack as it stands: a green CI run is currently doing silent work in the "PR state" check and in every reviewer's prior, and for Python agentic PRs it is close to uninformative about the diff. Whatever confidence the four gates leave to "well, the tests passed" is, on this measurement, mostly unearned.

For extending the stack: unlike a keyword deny-list, diff coverage is computed from the diff. It needs no calibration to a codebase's vocabulary, it cannot be disarmed by a change that reads clean, and it produces a number rather than a judgment — which puts it squarely in the deterministic tier alongside the size ceiling rather than behind the LLM veto. The obvious form is a floor ("auto-approve only if every added executable line is executed by some test"), and it composes with the size gate rather than duplicating it: PR size predicts how much risk, coverage predicts whether anything watches it. The cost is the one the paper cannot estimate — a coverage floor may shrink the auto-approvable set much harder than a path deny-list would, since more than half of Python agentic PRs would fail it outright.

Note that this measures the same open-source agentic-PR population as the security study, has the same missing piece (no human-authored baseline), and does not stratify coverage by PR size — so it cannot say whether small diffs, the ones StampHog actually auto-approves, are better or worse covered than the corpus average.

And the case study reports volume, not safety. 1 in 3 and 1.6K are throughput figures. P8/P9 is exactly this split — automated review reliably buys throughput and latency, its effect on quality and security is contested — and StampHog is a large deployed instance of the contested half with nothing measured attached to it. What would settle it is cheap and absent: the escaped-defect or incident rate of auto-approved PRs against the human-stamped baseline the channel used to produce.

Connections#

• Same-Model Review Blindness — a free, unclaimed tightening of the last gate. Three of the four checks are deterministic and the fourth is an LLM showstopper running last on a diff a coding agent produced; nothing in the design constrains which model runs it. Greptile's paired datasets (case-study) put 6–12 points of high-severity recall on that choice — each frontier model catches fewer bugs in its own family's code — so routing the veto to a different family than the authoring agent is worth more than any prompt tuning, costs nothing, and needs no calibration to the codebase the way the deny-list does. It is also the cheapest possible instance of the independence spec this same source states for its reviewer panel ("different models and providers for different reviewers"), applied to the gate rather than the panel. Note the ordering caveat: the veto sits behind three gates by design, so this raises the weakest layer without changing what reaches it

• Agent Review Comment Resolution — independent empirical support for two of this design's least-argued choices. Across 54,713 agent review comments, an inline code suggestion is the strongest predictor of resolution (OR 1.62, 75.5% vs 64.6%) while comment length carries a penalty concentrated in functional feedback (OR 0.855) — which is StampHog's minimal-output rule (a bare approval with no line comments; a refusal of 1-2 sentences plus a risk level and next steps) measured rather than asserted. And it validates demoting the model to a veto: a veto produces no comment, so it has no resolution rate to lose, and it sidesteps the failure that dominates that study's rejections — an agent flagging as a defect what the team decided on purpose (23.8% of argued cases). What it does not supply is the escaped-defect number this page's first open question asks for; adoption is not correctness there either

• Verification as the New Bottleneck — a partial, deployed answer to that page's "how far do you push fully automated reviews?": as far as the cheap, structural checks reach, with the model demoted to a veto behind them — not as far as a model's judgment about correctness. That page's CircleCI material (vendor-claim) is the same cheap-checks-first ordering pushed one stage earlier: run lint, unit tests, and builds in the inner loop so failures surface in seconds, and CircleCI models ~$700K/yr of a 50-developer team's delivery cost as recoverable that way. Both are ordering arguments — put the checks that are fast and certain ahead of the ones that are slow or probabilistic — differing only in where the cheap tier sits (the developer's loop vs the merge gate), and the composition is the obvious one: a diff that already passed the inner loop arrives at StampHog cleaner

• Review as the Control Point — the deployed instance of P17 (risk-tiered gating of material changes only), and a large uncontrolled instance of the contested P9: this reports throughput (1 in 3, 1.6K/month) and measures no quality outcome

• Security Debt of Agent-Generated Code — the empirical check on the gate stack, cutting both ways: the 16.2%→53.6% size gradient backs the diff-size ceiling, while 87.6% of smells living in CI/container files that match no business-risk keyword exposes the deny-list's blind spot, and the 18.9% credential-comment rate is the measured floor under the LLM showstopper check that would have to cover it

• Agent-Generated Test Quality — the rival gate predicate, measured (see the section above): existing tests execute 61.5% of agents' changed lines in Java and 27.0% in Python, and reach nothing at all in 64.8% of Python PRs — so "the tests passed" is not the reassurance the gate stack implicitly leans on. It is also the one candidate predicate that is computable from the diff, which makes it eligible for the deterministic tier where a keyword deny-list is a guess and the LLM check is a judgment

• Deterministic Pre-Execution Gates — the same deterministic-first ordering at the tool-call boundary, and the complement to this page's evidence gap. Two differences decide how they compose. StampHog's gates are proxies — a keyword deny-list and a line count stand in for "would a mistake here be expensive" — where a policy gate decides the actual question it was written for, which is why gate precision there is auditable (100% over 161 fires for one gate, 5% for another) and deny-list coverage here is not. And the reporting is mirrored: this case study measures volume and never efficacy, that paper measures efficacy (+12.4pp, replicated on disjoint seeds) and has no deployment. The transferable rule both support is the ordering, not either config

• Claude Code Auto Mode — the same classifier-gated-autonomy shape one layer down (per-tool-call rather than per-merge), with the same one-directional authority: the model may block, never unblock. The difference is what carries the decision — auto-mode's classifier is the gate, where here three deterministic checks gate and the model only vetoes

• Acceleration Whiplash — the lever against the review-time explosion (median time-in-PR-review +441.5%), and the mechanism by which review coverage can rise while human review-hours fall; but Faros's thesis is that the fix belongs in authoring, and a merge gate is downstream of that

• AI Brain Fry — the interruption side: what the gate removes is 1.6K context switches by engineers approving changes "they had little to no context on," and escalation-with-routing is the "concentrate review on high-stakes decision points" mitigation mechanized

• Optimizer–Evaluator Decoupling — the same source's reviewer-panel pattern applies the rule with an explicit independence spec: the agent that wrote the code can't review it, and reviewers should span different instructions, models, and providers

• Loop Engineering — the auto-stamper and the PR-babysitting loop are two loop-engineering products aimed at review-adjacent toil; PostHog's Paul D'Ambra puts a price on the discipline ("something like 60% of my token spend is burned automating the toil of handling CI and review")

• Blast Radius (Agentic) — a third sense of the term in the vault: here it is neither security-compromise scope nor code-change footprint but a merge-gate predicate — a keyword deny-list standing in for "would a mistake here be expensive," which is why the substitution's coverage is checkable and, per the security study, incomplete

• Unproductive Self-Verification — "verify by observation, not reasoning" is the reviewer-side statement of why an agent's account of its own work is the weakest available evidence; decomposing until each diff is runnable is the escape

• Polish No Longer Signals Readiness — what the deterministic gates route around: a size and path check cannot be disarmed by a diff that reads clean, which is the specific reviewer failure (P4) that a model-first gate inherits

• Write-Then-Trusted — the deny-list critique's second instance, one layer down: Pillar Security's verdict on allow-by-default sandbox profiles — "not a sandbox, it's a list of things somebody remembered to block that's always one entry short" — is the OS-policy version of this page's measured gap, where a blast-radius keyword deny-list misses the CI/container files carrying 87.6% of measured security debt. Two independent enumerate-the-bad controls, each falling short in the direction its author could not anticipate; the open question below ("does extending the deny-list restore coverage?") has the same shape in both

Derived#

• Is Human Review of AI-Authored Code Still a Real Control, or Already Rubber-Stamping? — the oversight synthesis this bears on: a deployed calibration of its Answer 3 partition, and a case where the automated layer replaced an approval ritual that was already rubber-stamping — so it changes the accounting of the merge stream more than it changes the safety of it

Open Questions#

• The case study reports volume and never efficacy. What is the escaped-defect or incident rate of auto-approved PRs versus the human-stamp baseline the Slack channel used to produce? PostHog has both populations in its own history, which makes this a checkable before/after rather than a request for new instrumentation. The same history answers a second question for free (added 2026-08-12): plotting the share of merged PRs that would clear the 500-line/20-file ceiling, quarter by quarter, dates how fast the gate's coverage decays as the ambient PR-size distribution rises.
• A keyword deny-list is a proxy for blast radius, and Security Debt of Agent-Generated Code shows the proxy misses where the measured debt concentrates (CI/container plumbing, 87.6%). Does adding CI/IaC paths to the deny-list restore coverage, or does it shrink the auto-approvable set so far that the gate stops paying for itself? A third candidate predicate is now priced but not tested (2026-08-12): Dipongkor et al. (empirical) show that "the existing suite covers this diff" is computable, deterministic, and rarely true — 27.0% of changed lines in Python, nothing at all in 64.8% of PRs. That reframes this bullet as a choice between two extensions with the same failure mode rather than one open question: both a path deny-list and a coverage floor buy precision by shrinking the auto-approvable set, and neither has been measured against the volume it costs. The same before/after PostHog already has in its history would answer both at once.
• If the size ceiling becomes a design target — agents instructed to emit stacks under 400 lines — does total risk fall, or does it just redistribute into more PRs each below the gate, with the integration risk moving to the seams between them? The security study's size gradient is measured per-PR and cannot distinguish these.

Sources#

• The State of AI Impact in Engineering: Q2 2026 — Justin Reock, The State of AI Impact in Engineering: Q2 2026 (DX, Engineering Enablement newsletter, 2026-07-22), tier corrected empirical to vendor-claim at compile (reasoning in Sources). Finding 2 only — median PR size nearly doubling Q1 to Q2 2026 across 500+ customer organizations — cited here as the moving distribution under this gate's fixed ceiling. No methodology accompanies the figure; the report is gated
• Stop being the code review bottleneck — Jina Yoon, "Stop being the code review bottleneck", PostHog newsletter (2026-07-09), case-study. §3 (StampHog: label trigger, four gates, invariants from the porting prompt, escalation routing, 1-in-3 and 1.6K figures, the #dev-stamp-exchange predecessor), §4 (observability over reasoning; stacked PRs under 400 lines via Graphite; the explicit composition with §3), §1 (reviewer panel, independence across models and providers, the 60%-token-spend quote)