LLM-Assisted Grey-Literature Theory Building

Sources#

• 3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse

Summary#

The "method paper in disguise" inside Agarwal, Miller, Kästner & Vasilescu (CMU, arXiv 2607.07980) — the secondary contribution behind their code-review theory. Grounded-theory building in software engineering is normally distilled from a small, deeply analyzed base (a few dozen interviews, a handful of teams, at most a few hundred screened papers). This paper scales it to thousands of practitioner documents by delegating the mechanical steps to LLMs while keeping the interpretive core with the researchers, and offers the pipeline (with a public replication package) as a template for future repository-mining and theory work.

The transferable result is a division of labor: LLMs are good at the labor-intensive, quote-anchored parts (corpus construction, relevance filtering, open coding); they are not good at the interpretive synthesis (axial + selective coding), and trying to automate that step produced garbage.

The pipeline#

1. Corpus construction (38,709 docs). Two registers, one AI-and-review lexicon. Reddit: the full 2020–2026 history of 43 subreddits in 7 strata via the public arctic shift archive, kept on a word-boundary "review" gate with no popularity filter (which would suppress the minority views thematic analysis needs) → 31,079 threads, each a Markdown doc with its comment tree. Web: edited long-form writing retrieved through the Exa neural search engine using 68 natural-language queries, each pairing a directional query with its opposite pole ("how AI is changing review" vs "why review hasn't fundamentally changed") to avoid tilting the corpus, over three passes (open web / full Feedspot blog ranking / category-balanced institutional domains) → 7,630 articles across 1,437 domains.
2. Relevance filtering (an LLM judge). A single neutral, versioned, reasoning-first rubric run by Gemini 2.5 Flash at temperature 0 over 23,631 candidates keeps 13,469 (57%); a stronger model re-judging a random sample agrees at Cohen's κ = 0.75 (substantial). Coding is then restricted to 2025–2026, when agent-authored PRs became a live concern.
3. Open coding (Thematic-LM). A source-stratified random sample of 3,100 documents is coded with Thematic-LM, a previously-evaluated multi-agent LLM realization of inductive thematic analysis whose codebook is built and revised as data arrive. Three coder agents attach 1–3 short codes (each with a verbatim quote) per segment; an aggregator merges near-duplicates; a reviewer maintains one versioned codebook via embedding similarity. Output: 4,838 codes grounded in 109,951 quotes from 2,669 documents. Cost ≈ US$0.35 per document across all steps.
4. Codes → theory (manual). The authors performed the axial and selective coding by hand over many iterations, using the model only to organize and retrieve evidence — querying the codebook as a search engine for the practitioner text bearing on a hypothesized construct or relationship, then confirming or revising it. Result: 26 constructs, 67 relationships, each grounded in coded text (contested edges carry opposing evidence too), with an append-only manifest tracing every quote back to its document via stable handles (e.g., G2935).

Bias management by design: three polarized coder lenses#

The sharpest methodological move. Rather than treating three coder agents as generic "perspective diversity," the authors instantiate three lenses for the polarized debate: a neutral inductive lens, a critical lens attentive to how agents/automated review may erode review, and an appreciative lens attentive to how they may strengthen it. Optimistic and pessimistic readings enter the codebook by design rather than by the model's default — a concrete answer to the worry that an LLM coder flattens a contested discourse toward its own priors.

The load-bearing negative result: automation fails at synthesis#

The single most transferable finding is where automation stopped working. Thematic-LM would normally develop themes in a second automated pass, but with nearly 5,000 codes that assumption breaks, and the step from codes to a causal theory is the interpretive heart of the study. To gauge how far automation could go, they extracted causal statements directly from the codebook in one bottom-up pass — it yielded 15,029 statements, but shallow, redundant, and overlapping ones, so they built the theory by hand instead.

Why it failed is the general lesson: practitioners use inconsistent terms, and the same term for different things. At one point a single "review rigor" construct conflated what are now review efficiency, review effectiveness, review depth, and reviewer skill — separating them took sustained interpretive work. "The distinction is genuinely useful but not inherent in the documents; we had to impose it by reading in context." Interpretation stayed with the human analyst. This is the same boundary the LLM-wiki pattern draws — the LLM does the bookkeeping; the synthesis/judgment is where a human stays load-bearing.

Honest limits#

• Sources are age-of-generative-AI text, and some may be partly or wholly LLM-generated; this studies discourse about review, read as practitioner argument, not verified firsthand practice.
• LLM coding/filtering can miscode, miss, or hallucinate — mitigated by three independent coders, a verbatim quote behind every code, a separately-validated relevance judge, and an author audit of every relationship, but residual error remains.
• The codes→theory step is author judgment at each structural step (by design — axial/selective coding is the interpretive core), so a different team could have drawn different constructs; grounding every one in coded text constrains but doesn't eliminate this.
• Grey literature over-represents vocal early adopters and carries vendor advocacy and post-incident hindsight — countered through the breadth of thousands of independent sources, not eliminated.
• Saturation was likely exceeded — the authors suspect far fewer than 3,100 documents would have sufficed ("we scaled because we could"), but because synthesis was manual they can't say where saturation set in.

Connections#

• Review as the Control Point — the theory this method produced (the paper's primary contribution)
• LLM-as-Compiler Knowledge Base — the same architectural boundary: an LLM compiles raw documents into a structured, interlinked, quote-grounded knowledge artifact, but the interpretive synthesis is where a human stays load-bearing; this vault runs the compile-side, this paper runs the theory-building-side of the same idea
• LLM-as-a-Judge — the relevance filter is a canonical LLM-judge deployment (neutral versioned rubric, Gemini 2.5 Flash, temp 0), used here as a corpus gate rather than an output grader
• LLM-Judge Validation — the paper practices exactly what that audit prescribes: it reports the chance-corrected Cohen's κ = 0.75 against a stronger re-judging model (not raw exact-match agreement) as the reliability number for its LLM judge
• The Solo-Authorship Rebound — the same execution/interpretation boundary, seen from population scale instead of from inside one pipeline. This page found by construction that LLMs absorb the mechanical, quote-anchored work and fail at interpretive synthesis; Matsui finds 300M+ OpenAlex works showing the shadow of that split across all of science — the tasks handed over are coding, data handling and statistical analysis, the recovered solo papers tilt measurably toward computational work (+0.040 s.d., P = 4×10⁻⁸), and what the solo author keeps is the part the CMU team also could not delegate
• Deep Research Agents — the corpus-quality risk this pipeline runs, measured. MisKnow-Agent (arXiv 2607.20891, empirical) generates the same false claim in four registers — paper, news, blog, forum post — and finds that genre outweighs provenance as an adoption cue for the reading model: a 23.5pp paper-to-post gap against 14.8pp from high- to low-authority institution, with the generation protocol explicitly holding writing quality constant across authority tiers so the authority effect is the name and URL alone. Two consequences for grey-literature theory building. The register mix is not neutral — a pipeline sampling Reddit, blogs and institutional long-form is weighting a signal that turns out to be cheap to fake, and the polarized-lens design guards against coder bias, not against a persuasive-but-wrong document entering the codebook in the first place. And it argues for keeping the interpretive step human on a second ground beyond the shallow-causal-statements failure: the automatable half of the pipeline is exactly the half that reads format as credibility

Open Questions#

• The authors couldn't locate the saturation point because synthesis was manual — how few documents actually suffice, and can a cheaper sample match the 3,100-doc theory?
• Automating the codes→theory step failed with a naïve bottom-up prompt; is that a prompt/scaffolding limitation or a genuine ceiling on LLM interpretive synthesis over thousands of codes?
• The three-lens design manages coder bias, but the relevance judge and segmenter are single-model — do those upstream gates impose their own systematic slant on what reaches the codebook?

Sources#

• 3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse — Agarwal, Miller, Kästner & Vasilescu (CMU, arXiv 2607.07980, 2026-07-08), empirical. §III (methods: corpus construction, relevance filtering, coding, codebook→theory), §III-A.1 (limitations), §VI (benefits/limits of scaling grey-literature analysis, the 15,029-statement negative result, saturation), Appendices 2–5