Production-Sourced Evaluation

Sources#

• DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity
• Driving the Agent Quality Flywheel from Your Coding Agent- Google Developers Blog
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
• The price is wrong: AI cost calculation has to consider task completion rates, not just token costs

Summary#

Production-sourced evaluation builds a benchmark from real, de-identified usage of a deployed system rather than from synthetic generation or hand-authored prompts. The argument: a benchmark's whole value is predicting real-world performance, and the most representative tasks are the ones users actually issued. DRACO (Perplexity, 2026) is the worked example — its 100 deep-research tasks are distilled from tens of millions of real Perplexity Deep Research queries — and the method is the paper's central contribution, distinct from the rubric design or the grading protocol.

The method (DRACO)#

1. Difficulty-proxied sampling. Start from production traffic, biased toward hard cases: DRACO sampled 1,000 queries that drew subsequent negative sentiment or an explicit thumbs-down — i.e., the queries the deployed system handled worst. This mines failures the system already exhibits, which synthetic generation can't target.
2. Privacy-preserving reformulation. An automated LLM pipeline strips PII and reduces ambiguity. Critically, no raw user query is ever exposed to a human analyst — anonymization is a precondition, enforced architecturally, not a cleanup step.
3. Augmentation toward difficulty + specification. Real queries are often under-specified; augmentation adds context (persona, output format, sources) and broadens scope (temporal, comparative, geographic) so tasks become well-defined and challenging while still reflecting implicit user intent.
4. Filtering for objective / tractable / difficult. Keep only tasks with convergent expert success criteria, bounded scope, and genuine difficulty.
5. Human gate. A final in-house expert review for security and quality. The pipeline is automatable end-to-end but deliberately keeps a human as the last safety/quality gate.

The payoff DRACO claims: a benchmark that is representative (mirrors the real domain mix and real failure modes) and refreshable (because both research needs and usage evolve, the pipeline can regenerate fresh tasks rather than ossifying).

The core tradeoff: representativeness vs. over-specification#

Production-sourcing buys representativeness but the augmentation step that makes raw queries evaluable also threatens it. The paper is candid: systematic augmentation "reduces ambiguity and improves reproducibility, but it also risks over-specifying tasks and dampening the natural variability of user queries." The de-identification + augmentation pipeline turns a messy, personal, ambiguous query into a clean, bounded, comparable task — and some of what's stripped (ambiguity, personal context, the actual phrasing) is also part of what makes real usage real. Production-sourced is more representative than synthetic, but it is not raw production.

And representativeness of the tasks is orthogonal to validity of the grading: a benchmark can mine exactly the right production queries and still report an untrustworthy verdict if the judge scoring them hasn't been validated. Norman et al. (2026) is the other half — chance-correction, position-swap, and the consistency–bias audit on the grader — so a fully trustworthy production-sourced eval needs both a representative task distribution and a validated judge.

Why production traffic is a moat-grade eval asset#

The method only works if you have a large-scale deployed system generating the traffic — which is exactly the proprietary-data position described in Compounding Data Moat. Real usage at scale is "time-locked, context-specific, and impossible for a copycat to recreate"; here that same asset doubles as an evaluation substrate. A vendor with production traffic can build representative, difficulty-targeted, continuously-refreshed benchmarks that a competitor without deployment simply cannot — and can do so on tasks where its own product currently fails (the thumbs-down sampling). This is the data flywheel pointed at measurement: usage → failure signal → benchmark → product improvement.

The flip side is a credibility question (see DRACO Benchmark): a benchmark sourced from one vendor's traffic, on which that vendor's product wins, carries an obvious incentive — the human gate and the expert rubrics are partly there to answer it.

The product-loop form: Google's flywheel#

Google's Agent Quality Flywheel operationalizes the same principle as a continuous product loop rather than a benchmark. Agents emit OTel traces; each production session "is a genuine request… and each failure is a ready-made test case for the next cycle." Complete traces skip inference and are graded in place; Online Monitors score live traffic continuously, and drifting scores hand failing traces to the eval-fix loop. Google states the ordering explicitly: synthetic scenarios (its User Simulator) are a cold-start bootstrap — "synthetic scenarios get you moving; production data is what makes the loop sharp." That makes three independent arrivals at production-as-eval-substrate: DRACO (capability benchmark), Deployment Simulation (safety forecasting), and the flywheel (continuous quality monitoring) — the method crossing from benchmark construction into day-to-day product tooling.

The buyer-side instance: Databricks builds its own (2026)#

The method arriving from the fourth direction — not a benchmark vendor, not a product loop, but a customer building an eval to decide what to buy. Databricks reports, via The Register (2026-07-13, case-study, secondary reporting), an internal coding benchmark devised from real engineering tasks its own staff performed against its multi-million-line codebase. The stated motivation is contamination-by-optimization rather than representativeness: CTO Matei Zaharia says the company ran the evaluation because models are tuned to existing benchmarks like SWE-Bench, which the article notes OpenAI has called "broken." The eval's output was a purchasing decision — per-task cost and success rate by model and by harness (Cost-per-Task Over Cost-per-Token, Orchestration Sets Token Economics) — which is exactly the tie-breaker role the vendor advice above assigns to it.

Two things separate it from DRACO's pipeline. It is production-sourced tasks, not production traffic: staff engineering work, not de-identified user queries, so no privacy pipeline is needed and no thumbs-down difficulty proxy is available. And its representativeness claim is stated at company scope, not domain scope — Zaharia concedes the results reflect Databricks' own codebase while arguing other companies can run the same evaluation against theirs. That is the honest form of the moat argument on this page: the asset is not transferable, but the method is, and a buyer with a large codebase already owns the substrate.

Contrast with the alternatives#

• Synthetic generation (DeepResearchEval, ReportBench, DeepScholar-Bench, DRBench) — scalable, no privacy exposure, but tasks are model-imagined and may miss real failure modes.
• Hand-authored from interviews/searches (xBench, ResearcherBench, DEER) — human-authored and realistic, but bounded by author imagination and not drawn from a live production system.
• Production-sourced (DRACO) — the only one of the three that mines the actual distribution and the actual failures, at the cost of needing deployment access and a privacy pipeline.

Connections#

• DRACO Benchmark — the worked example; this method is its central contribution
• Automated Failure Attribution — the opposite trade, taken deliberately and worth reading as the counterweight to this page. WHO&WHEN PRO reaches 12,326 failure traces with golden agent/step/mode labels precisely by synthesizing them — one error injected into a warm-started run that had already succeeded, so the decisive step is correct by construction rather than by annotation. Nothing production-sourced can match that label fidelity at that scale; nothing synthesized can speak to what actually fails in deployment or how often. The two failure modes are exact mirrors: a production-sourced eval has real tasks and contestable labels, a synthesized one has perfect labels and a designed distribution
• Cost-per-Task Over Cost-per-Token — the same method as vendor advice: Anthropic's model-selection guidance concedes that public benchmarks saturate at the Opus/Fable tier and tells buyers to curate the deciding eval from their own production traffic instead — production-sourced evaluation as the tie-breaker for a purchasing decision, not just a benchmark-construction technique. Databricks is that advice executed by a buyer, and the numbers it produced (per-task cost and success rate by model) live on that page
• Orchestration Sets Token Economics — the other half of what a buyer-built eval measured: with the tasks drawn from its own codebase, Databricks could vary the harness as well as the model, which no public coding benchmark exposes. Sourcing the tasks from real engineering work is what makes a cross-harness cost comparison mean anything
• LLM-as-a-Judge — the grading half of the pipeline; production-sourced tasks + rubric-judge grading make an automatable (human-gated) eval
• Deep Research Agents — the system class whose production traffic DRACO mines
• Compounding Data Moat — production usage as a time-locked proprietary asset; this is that asset repurposed as an evaluation substrate
• Evals as Product Spec — "build your measurement framework before launch / from real usage"; production-sourced evaluation is that principle at benchmark scale
• Task Time-Horizon Scaling — sibling concern: benchmarks saturate, so the ability to refresh from live usage is what keeps an eval alive
• Automated Behavioral Audit — the alignment-side analog notes its synthetic scenarios "may not match real-traffic distributions" — exactly the gap production-sourcing closes
• Telemetry vs. Survey Measurement — Faros AI's telemetry-over-survey stance is the engineering-metrics sibling: measure from the real system, not from self-report
• Deployment Simulation — the alignment-side application of the same method: OpenAI replays de-identified production conversations to forecast safety behavior pre-release, where DRACO replays them to build a capability benchmark; same PII pipeline, same proprietary-traffic moat
• Conversation-to-Delegation Shift — its measurement-obsolescence argument is the same instinct one step further: as usage becomes delegation, even which metrics to read (complexity, runtime, concurrency, output) must be re-sourced from real agentic behavior, not interaction counts
• Agent Quality Flywheel — the continuous product-loop form: OTel production traces graded in place, Online Monitors on live traffic, synthetic simulation demoted to cold-start bootstrap
• Failures That Look Like Success — the failure class production-scale traces could quantify: silent contract violations that demo-sized evals only sample
• Context Advantage, Not Taste — production telemetry as context transfer: sourcing evals from real usage moves what the human knows about users into where the model can read it, spending the human's asymmetry by design
• LLM-Judge Validation — the orthogonal quality axis: this page fixes which tasks the benchmark contains; judge validation fixes whether the grading of them is trustworthy — a representative task graded by an unvalidated judge is still an unreliable eval
• Measuring Beyond Accuracy Saturation — the sibling answer to "what to do when a benchmark saturates," from the opposite end: this page refreshes the task set from live production usage; Nadgir et al. re-instrument the existing task set along six non-accuracy axes. Both reject the retire-and-replace reflex — new tasks vs new metrics
• Benchmark Contamination and Decontamination — the prevention vs correction pairing against data contamination: production-sourcing (and dynamic benchmarking generally) avoids leakage up front by drawing fresh, hard-to-pre-memorize tasks and refreshing them; UBD instead repairs a model already inflated by exposure to a static benchmark, without a clean reference. Complementary defenses against the same leakage threat
• How Much Signal Do Public Benchmarks Still Carry — and What Replaces Them? — the cluster synthesis: production-sourced refresh is the new-tasks move of the five-part replacement portfolio (contamination prevention + representativeness), paired there with judge validation as the two halves of a trustworthy eval

Open Questions#

• How much does augmentation distort the distribution it claims to represent? Is there a measurable representativeness loss between raw queries and augmented tasks?
• Difficulty-by-thumbs-down biases toward current failures — does that make the benchmark a moving target that flatters the next model trained on those failures?
• Can the privacy pipeline (no human sees raw queries) be trusted/audited well enough for regulated domains (medicine, law) where the source traffic is most sensitive?

Sources#

• DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity — §3 (task construction: sampling, pre-processing, augmentation, filtering, curation), §6.1 (generalization limits; augmentation over-specification caveat)
• Driving the Agent Quality Flywheel from Your Coding Agent- Google Developers Blog — "From the inner loop to the production loop": OTel traces as eval input, Online Monitors, synthetic-as-bootstrap (vendor-claim)
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias — Norman et al. (arXiv 2606.19544, June 2026, empirical): the grading-validity half of eval quality, orthogonal to task representativeness; see LLM-Judge Validation
• The price is wrong: AI cost calculation has to consider task completion rates, not just token costs — Thomas Claburn, The Register, 2026-07-13 (case-study, secondary reporting of Databricks' benchmark blog post; the primary is not in the corpus): the buyer-side instance — an internal coding benchmark devised from staff engineering tasks against a multi-million-line codebase, motivated by models being tuned to SWE-Bench, with the results driving model and harness selection