Agent Harness Engineering

Sources#

• An open-source spec for Codex orchestration: Symphony.
• Anthropic's Boris Cherny: Why Coding Is Solved, and What Comes Next
• Effective harnesses for long-running agents
• Full Walkthrough: Workflow for AI Coding — Matt Pocock
• Harness engineering: leveraging Codex in an agent-first world
• How Anthropic's product team moves faster than anyone else | Cat Wu (Head of Product, Claude Code)
• Tips & Best Practices
• Tutorial: Team Telegram Assistant

Summary#

Agent harness engineering is the discipline of designing environments, artifacts, and feedback loops that enable AI coding agents to do reliable, sustained work across multiple context windows. The core shift: the engineer's job moves from writing code to building the scaffolding that makes agents effective — specifying intent, structuring context, enforcing invariants, and constructing verification pipelines.

Details#

The Fundamental Problem#

AI coding agents work in discrete sessions with limited context windows. Each new session starts with no memory of prior work. Without deliberate harness design, agents exhibit predictable failure modes:

1. One-shotting — attempting to build everything at once, exhausting context mid-implementation, leaving half-finished undocumented work
2. Premature victory — seeing partial progress and declaring the job done
3. Dirty state — leaving the environment with bugs, uncommitted changes, or undocumented progress for the next session to untangle
4. Incomplete verification — marking features as complete without end-to-end testing

Two-Agent Architecture (Anthropic)#

Anthropic's solution for the Claude Agent SDK uses two specialized prompts:

• Initializer agent (first session only): scaffolds the environment — writes an init.sh script, creates a claude-progress.txt log, generates a structured JSON feature list with all requirements marked as "failing," and makes an initial git commit.
• Coding agent (every subsequent session): reads progress logs and git history, runs a basic smoke test, picks a single feature to implement, verifies it end-to-end (e.g., via Puppeteer MCP for web apps), commits clean state, and updates the progress file.

The JSON feature list is critical: agents are instructed never to remove or edit feature descriptions, only to flip passes from false to true after verification. JSON was chosen over Markdown because agents are less likely to accidentally overwrite structured JSON.

Repository as System of Record (OpenAI)#

OpenAI's Codex team built a product with zero manually-written code (~1M lines, ~1,500 PRs, 3–7 engineers over 5 months). Their key architectural insight: repository-local, versioned artifacts are all the agent can see — anything in Slack, Google Docs, or people's heads is invisible.

Their approach:

• AGENTS.md as table of contents, not encyclopedia: a short (~100 lines) map pointing to deeper docs. A monolithic instruction file fails because it crowds out task context, becomes non-guidance when everything is "important," rots instantly, and resists mechanical verification.
• Progressive disclosure: agents start with a small stable entry point and are taught where to drill deeper. Design docs, execution plans, and technical debt are all versioned in-repo.
• Mechanical enforcement: custom linters (themselves agent-generated) enforce architecture — dependency direction between layers, structured logging, naming conventions, file size limits. Lint error messages are written as remediation instructions injected into agent context.
• Doc gardening: a recurring background agent scans for stale documentation and opens fix-up PRs.

Enforcing Architecture at Scale#

Both sources converge on a key principle: enforce invariants, not implementations. Define strict boundaries (layer dependencies, data validation at boundaries, naming conventions) and let agents have freedom within those boundaries.

OpenAI uses a rigid layered architecture per business domain: Types → Config → Repo → Service → Runtime → UI, with cross-cutting concerns entering through a single Providers interface. This is enforced by structural tests and custom linters. They note this level of architectural rigor is usually postponed until hundreds of engineers — with agents, it's an early prerequisite because constraints enable speed without drift.

Continuous Entropy Management#

Agent-generated codebases accumulate entropy: agents replicate patterns that already exist, including suboptimal ones. OpenAI initially spent 20% of engineering time on manual "AI slop" cleanup. Their solution: encode "golden principles" into the repo and run background agent tasks on a recurring cadence to scan for deviations, update quality grades, and open targeted refactoring PRs. This functions as garbage collection — paying down technical debt continuously in small increments rather than letting it compound.

Harness as Service#

The patterns above describe per-session harnesses. Two 2026 systems demonstrate the natural evolution — harnesses that run continuously as services with per-tenant workspace isolation:

• Symphony (OpenAI, March 2026) — long-running daemon polling Linear, per-issue workspace, Codex App Server session per ticket. Same team that authored the OpenAI source above; their explicit "harness as service" iteration. Orchestrator owns workspace lifecycle, retry/backoff, stall detection, and reconciliation; in-repo WORKFLOW.md is the policy file.
• Hermes Agent (Nous Research) — Hermes Gateway runs as systemd or launchd; per-user session isolation; allowlist + DM-pairing authorization; cron jobs delivered to a designated home channel.

Convergent design choices across the two:

1. Daemon-first deployment — long-running service, not per-invocation CLI.
2. Per-tenant workspace isolation — per-issue (Symphony) or per-user (Hermes).
3. Container backends as the trust boundary (Docker, Singularity, Modal, Daytona) rather than per-command approval prompting. Hermes explicitly disables dangerous-command checks under a container backend on the principle that "the container is the security boundary."
4. Repo-versioned markdown as control plane — WORKFLOW.md for Symphony, AGENTS.md/SOUL.md for Hermes, same pattern as the CLAUDE.md/AGENTS.md table-of-contents discipline at session level.
5. No durable orchestrator DB by default — Symphony explicitly chooses tracker + filesystem for restart recovery; Hermes Gateway state is filesystem-only.

Symphony's evolution sharpens the principle stated above. Their first version treated agents as rigid state-machine nodes — Codex was only asked to implement the task in a ticket. They found this too limiting once models grew capable enough to "create multiple PRs as well as read review feedback and address it," and shifted to giving agents objectives + tools, not state transitions. This is "enforce invariants, not implementations" applied at the orchestration layer (see Ticket-Driven Agent Orchestration).

For the integration boundary between orchestrator and coding agent, Symphony exercises the Codex App Server protocol — JSON-RPC over stdio with continuation turns and dynamic tool calls — which makes the contract explicit and version-tolerant. The protocol's dynamic tool calls feature is also a notable harness primitive: orchestrator-implemented tools can wrap credentials the subagent should never see (e.g., Symphony's linear_graphql tool proxies authenticated GraphQL without giving subagent containers the Linear access token).

The Role of the Human#

In both systems, humans work at a different abstraction layer:

• Prioritize work and translate user feedback into acceptance criteria
• Design environments and feedback loops
• Validate outcomes and provide taste/judgment
• When the agent struggles, diagnose what capability is missing (tools, guardrails, documentation) and feed it back into the system — always via the agent, not by writing code directly

Connections#

• LLM-as-Compiler Knowledge Base — shares the pattern of repository-local knowledge as system of record, incremental compilation, and LLM-maintained artifacts
• Claude Code Best Practices — practical application of many harness engineering principles in Claude Code's environment (CLAUDE.md, skills, hooks, subagents)
• LLM-Driven Vulnerability Research — the vulnerability-finding scaffold is a minimal harness: isolated container, single prompt, agentic loop with file-ranking pre-pass and validation agent
• Client-Side Agent Optimization — harnesses provide the execution substrate that client-side optimizers then tune via combo selection; the invariants a harness enforces constrain the space AgentOpt searches over
• Scale-Dependent Prompt Sensitivity — output-length invariants (via system prompts, schemas, or validators) are a harness-level mitigation for scale-dependent overthinking — fits the "enforce invariants, not implementations" principle
• Claude Code Auto Mode — classifier-based tool-call gating is a concrete instance of "enforce invariants mechanically" at the permissions boundary — destructive-action limits enforced pre-execution rather than via advisory prompt
• Claude Opus 4.7 — better filesystem-memory reinforces the case for repository-local versioned artifacts as agent memory; task budgets echo the discipline of explicit resource envelopes that harnesses already impose
• Symphony — the natural "harness as service" evolution from the same OpenAI team; ticket-as-unit and per-issue workspace are direct extensions of the harness patterns established here
• Ticket-Driven Agent Orchestration — orchestration-layer restatement of "enforce invariants, not implementations"; once the per-session harness works, the next bottleneck is which session runs next
• Codex App Server Protocol — the integration boundary that makes "harness as service" possible; orchestrator drives sessions through a versioned JSON-RPC contract instead of scraping a CLI
• Hermes Agent — parallel daemon-first agent ecosystem; per-user instead of per-issue isolation, with the same container-backend safety pattern; bounded MEMORY.md/USER.md files implement explicit memory envelopes
• Context Window Smart Zone — the underlying constraint motivating system-prompt minimalism, AGENTS.md-as-ToC, and reviewer-in-fresh-context discipline; quadratic attention scaling sets the budget every harness operates within
• Agent Loop Pattern — the natural session-level primitive once the per-session harness works: drain a Kanban backlog AFK, fragment work into many fresh-context iterations
• Vertical Slice Tracer Bullets — planning-layer restatement of "enforce invariants, not implementations" — invariant is "every slice produces visible feedback"
• Design Concept Grilling — alignment-layer harness primitive; prevents premature plan generation by forcing pre-plan interview
• Deep Modules for Agents — codebase-shape complement: agents in deep-module codebases conserve smart-zone tokens and have natural test boundaries
• Harness Shrinkage as Models Improve — division-of-labor between harness and model: prompt scaffolding shrinks with model improvements, mechanical verification stays load-bearing
• Model Introspection Feedback — debugging-time tool: ask the model why it failed, fix the harness, not the model
• Interaction Models — resolves the harness-vs-model question firmly toward the model for the interaction layer (real-time A/V); VAD / turn-detection / dialog-management harnesses dissolve into model behavior (Thinking Machines Lab, May 2026)
• The Bitter Lesson — the principle behind "enforce invariants, not implementations": don't hand-engineer what scaled general capability will subsume
• Interaction / Background Model Split — the same multi-agent split, but for temporal concerns (stay responsive vs. think hard) rather than context isolation; an interaction-layer instance of the harness-vs-model division of labor
• MCP and Computer Use — connectors are the non-harness substrate; the model decides which one to use; harness logic shrinks around tool-dispatch decisions as the model picks better
• Agent Context Files — the advisory half of "enforce invariants, not implementations": CLAUDE.md/AGENTS.md/WORKFLOW.md are the policy plane; hooks and orchestrator invariants are the mechanical half

Derived#

• Opus 4.6 → 4.7 Changes and Multi-Agent Coding Considerations — applies "enforce invariants, not implementations" and the Writer/Reviewer pattern to an Opus 4.7 multi-agent coding team

Connections#

• Agent-Native Infrastructure — building agent-legible environments is harness engineering at the infrastructure layer; Karpathy's "describe it to agents first" is the same instinct as AGENTS.md-as-table-of-contents
• AI-Driven Formal Proof Search — the AlphaProof Nexus proof-sketch-with-EVOLVE-BLOCK-markers is a harness that "enforces invariants, not implementations" for theorem proving (the agent may edit only marked regions; the theorem statement is invariant)

Open Questions#

• Does a single general-purpose coding agent outperform a multi-agent architecture with specialized testing, QA, and cleanup agents?
• How does architectural coherence evolve over years in a fully agent-generated system?
• At what codebase scale does the AGENTS.md-as-table-of-contents approach need to be replaced with more sophisticated context routing?
• How generalizable are these web-app-focused findings to other domains (scientific research, financial modeling)?

Sources#

• Harness engineering: leveraging Codex in an agent-first world
• Effective harnesses for long-running agents