Harness Build-vs-Buy

Sources#

• Coding Agents and Technical Debt
• The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI

Summary#

Every organization that decides it needs "our own coding agent" is making a make-or-buy decision, usually without a price. Rajiv Shah (OpenHands, July 2026) supplies the first one in this corpus: twelve months of public GitHub activity across four coding-agent codebases — OpenHands (4 repos summed), OpenAI Codex, OpenCode, and Hermes.

Product	Merged PRs, 12 mo	Bug-fix PRs	Changed lines	Current code size
OpenHands (4 repos)	5,679	1,778 (31%)	~2.4M	~1.05M lines
OpenAI Codex	7,688	1,202 (16%)	~3.8M	~1.32M lines
OpenCode	5,700	2,304 (40%)	~2.9M	~1.05M lines
Hermes	7,736	5,288 (68%)	~3.0M	~1.75M lines

The convergence Shah flags is the interesting part: OpenHands and OpenCode were built by different teams in different languages with different architectures, and land 21 merged PRs apart at the same code size — 5,679 vs 5,700, ~1.05M lines each. He offers that as "a solid estimate of the work behind a production coding agent," which is the closest thing the corpus has to a price tag on the whole category.

Evidence note. case-study, and the interest is not incidental: OpenHands sells the maintained agent, and the post's conclusion is "don't fork, don't build in-house." Treat the two halves differently. The counts are verifiable public data with the method disclosed (GitHub GraphQL/REST, pulled 2026-07-08, bots excluded) and they are internally consistent — the four OpenHands repos sum exactly to the headline 5,679 PRs and ~1.05M lines. The interpretation — churn equals inherited debt equals don't build — is vendor-favorable framing and is attributed inline throughout below. Shah's own caveats travel with the numbers: bug-fix classification is label-heuristic ("compare the shares directionally"), OpenHands is summed across four repos while the other three use one flagship repo each (so their totals are understated), and code size excludes lockfiles, generated code, vendored deps, and binaries.

The bug-fix spread is not a quality signal. The band is 16% (Codex) to 40% (OpenCode) with OpenHands at 31%; Hermes' 68% is flagged in the article itself as probably a labeling-convention artifact ("although its labeling conventions may explain some of that difference"). Quote 68% only with that caveat attached; the comparable band is 16–40%.

A coding agent is never one codebase#

Shah's structural claim: "'A coding agent' sounds like one program. In practice, it is a platform." OpenHands makes the split legible because its pieces are separate public repos:

Piece	What it is	Code size	Merged PRs, 12 mo
OpenHands app	Agent application and server	404K lines	2,600
Software Agent SDK	Agent runtime	333K lines	2,036
Agent Canvas	UI for working with agents	246K lines	719
OpenHands CLI	Terminal interface	67K lines	324

The others have the same shape inside one repo: Codex holds "the CLI, dozens of core runtime crates, an app server, MCP support, and Python and TypeScript SDKs"; OpenCode's packages/ has core, server, cli, desktop, tui, web, two SDKs, and a plugin system. This is Shared Harness, Differentiated Surfaces observed from the repository side and from a third vendor — one runtime underneath, several surfaces on top — except here the split is being priced rather than argued for.

The pace is rising, not falling. Codex went from ~124 merged PRs/month in mid-2025 to ~1,000/month a year later; Hermes went from a near-empty repo to ~2,000 merged PRs/month "within months of getting serious."

The fork delta#

The argument's sharp end is a counterfactual, not a projection. Fork the OpenHands app twelve months ago, never merge from upstream, and today you are:

• 2,600 merged PRs behind on the app alone, including 866 bug fixes you do not have — "crashes, regressions, security patches, and model-compatibility repairs found by 179 contributors while your team worked on custom features"
• plus the 2,036 PRs the SDK underneath moved in the same window
• more than 4,600 PRs total, "about thirteen a day, every day, for a year"

Shah's three exits, all bad: rebase (conflict resolution against a year of changes to the same core files your customizations touch, "and the work returns with the next rebase"), cherry-pick ("someone reads a dozen-plus upstream PRs every day and decides which ones apply — that is a standing job"), or go it alone (you now own 5,000–8,000 merged PRs/year across 1M+ lines).

What this does not establish. The figure prices tracking all of upstream, not the value of the delta. A fork that deliberately freezes a narrow surface — no web UI, no multi-provider support, one deployment target — inherits only the fraction of upstream churn touching what it kept, which could be a small minority of the 4,600. Shah concedes exactly this in one unquantified clause ("your internal version may need only a fraction of that surface area, but even then it's still a lot of work") and moves on. That concession is load-bearing: the whole build-vs-buy calculus turns on the size of that fraction, and 13 PRs/day is an upper bound for a full-surface fork rather than an estimate for a narrow one.

The customization ladder#

The prescription is a ladder — customize at the highest layer that works, and only descend when it doesn't:

1. Configuration and system prompts. "A surprising amount of 'we need our own agent' turns out to mean 'we need our own prompt, tools, and defaults.'" See Agent Context Files.
2. MCP servers for internal systems. A tool server keeps the integration outside the agent, so upgrading the agent doesn't mean reapplying it to a fork. See MCP and Computer Use.
3. Skills or plugins for repeatable work — "your organization's expertise without creating a permanent fork delta." See Agentic Work Systematization.
4. Build on the SDK when you genuinely need a different agent (OpenHands SDK, Claude Agent SDK): "your team owns the product-specific layer while upstream keeps maintaining the runtime underneath it. For OpenHands, that means the 2,036 annual SDK PRs keep happening upstream instead of becoming your backlog."

Rung 4 is Shared Harness, Differentiated Surfaces sold as a buy recommendation: OpenAI built one harness and forked the surfaces internally; Shah's pitch is that an enterprise should rent the harness and own only its surface. And if you build anyway: "assign several engineers to it and make upstream tracking part of their job. Treat it as a product, because that is what it becomes."

The tension with harness shrinkage#

This is the load-bearing contradiction, and it mostly dissolves — but not entirely, and the residue is worth logging.

Harness Shrinkage as Models Improve holds that the harness shrinks toward a residue as models improve: Anthropic deleted >80% of Claude Code's system prompt for Claude 5 models with no measured eval loss (and a mild gain under SIMPLE=1), and Boris Cherny predicts Claude Code "may be 100 lines of code a year from now." Shah's data says four coding-agent harnesses are million-line codebases absorbing thousands of merged PRs a year. Three reasons both hold:

1. Prompt is not codebase. Every shrinkage measurement in the corpus is taken on the system prompt — the wiki's own resolved question restates the thesis as "request shrinkage." Deleting 80% of a prompt removes zero lines from an app server, a sandbox runtime, a desktop UI, or an SDK.
2. Cherny's residue is exactly what a million lines looks like. His answer to "what's left" is "almost all of it is about safety and permissions and static analysis and there's a bunch of UI code." That is not a small program, and OpenHands prices it: Agent Canvas (246K) plus CLI (67K) is 313K lines of pure surface before the server or the runtime.
3. Churn is not size, and shrinkage produces churn. Cat Wu's read-the-whole-prompt-at-every-launch pass, Cherny's line-by-line ablation, and re-validating vision workarounds against each release are themselves merged PRs. Shah agrees from the other direction: "your team is signing up to track every model release and agent design shift, indefinitely." A faster model cadence raises maintenance flow while lowering steady-state prompt size. The two theses predict opposite signs only if you assume PR count tracks line count.

What doesn't dissolve. Cherny's "100 lines" is a claim about the harness codebase, and this is the first measurement in the corpus pointed at that variable — for other people's harnesses, on a directional trajectory that is up: Codex ~124 → ~1,000 merged PRs/month over twelve months. That is not a falsification (the prediction is about Claude Code, testable in 2027, and Cherny's own framing concedes the literal number is hyperbole), but it is the first counter-datum against the reference class, and it is case-study public data against a self-labeled prediction. Recorded against that open question on Harness Shrinkage as Models Improve.

The same shape reappears in Agentic Loops Overtake Bespoke Systems, which reaches "don't build the bespoke thing" by the capability route — DeepMind's hand-built proof-search apparatus collapsed to a cost difference as the LLM improved. Shah reaches the same conclusion by the maintenance route, and the routes are independent: his argument would hold even if a fork never lost a single capability, because what it loses is 866 bug fixes a year.

The other vendor's answer: own it, because it is the P&L#

Shah prices the cost of owning a harness. Writer's harness-swap paper (The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI, arXiv 2607.06906, 2026-07-08, empirical) prices the upside of owning a good one, and reaches the opposite recommendation from the same starting question.

Its measurement: hold 22 tasks, six models, one judge panel and one price table fixed, swap only the orchestration layer, and cost per task falls 41% with tokens down 38% at quality parity — every model cheaper by 33–61%, no exceptions (Orchestration Sets Token Economics). The economic argument built on that is specifically about ownership: a harness improvement multiplies against every model in the fleet simultaneously and keeps multiplying when the model set changes, because it sits above the model API. At one million agent tasks a month that is $210k versus $120k — $1.08M a year from an orchestration change alone. Hence: "An organization that rents its orchestration layer has outsourced the variable it controls most."

	Writer	OpenHands (this page)
Recommendation	own the orchestration layer	rent the runtime, own only your surface
Evidence tier	empirical (controlled swap)	case-study (public GitHub counts)
What it prices	the upside of a good harness	the maintenance cost of keeping one
What it omits	the maintenance bill	the size of the efficiency delta at stake
Vendor interest	sells an orchestration platform	sells a maintained agent runtime

Both are vendor-authored and both sell exactly the conclusion they reach — Writer's COI is total (33 authors, all employees; the harness, the baseline and one of the six models are all theirs). Neither is wrong about its own half, and neither prices the other's: Writer never asks what its harness costs to maintain against a rising model cadence, and Shah never asks what the fork's efficiency delta is worth. The joint reading is that orchestration is high-leverage and expensive to own, which is an argument for the customization ladder above rather than for either extreme — and specifically for its rung 4, where you own the layer that sets your token bill while someone else absorbs the runtime's PR flow.

One genuine narrowing: Writer's claim is about the token-economics layer (context assembly, cache shape, compaction, offload, failure-spend governance, per-task accounting), not about the whole platform. Shah's counts cover app servers, desktop UIs, CLIs and SDKs, which is where most of the 1M+ lines and most of the churn live. Owning your cost policy is not the same commitment as owning your agent, and the two arguments are much closer to compatible once that distinction is drawn than either author draws it.

Which technical debt this is#

The article is titled "Coding Agents and Technical Debt," but the mechanism is not Agentic Technical Debt's. That page's debt is intent drift inside AI-written application code — each session re-deriving architecture, an inconsistency between compromises. This is classic vendor-branch divergence: the debt is the growing delta between your fork and a codebase someone else is actively improving, and it accrues from other people's velocity rather than your own shortcuts. The two share a maturity profile (invisible until a forced rebase or a forced rewrite) and nothing else — do not read the counts here as evidence about agent-authored code quality.

Connections#

• Orchestration Sets Token Economics — the opposing recommendation with the opposite half of the ledger measured: a controlled orchestration swap worth −41% cost and −38% tokens across six models, from which Writer concludes that renting your orchestration layer means outsourcing the variable you control most. Both vendor-authored, both selling their own conclusion; the narrowing that makes them compatible is that Writer's claim covers the token-economics layer, not the app server, UI and SDKs where this page's churn actually lives
• Shared Harness, Differentiated Surfaces — the same one-runtime-many-surfaces architecture, priced from the repo side by a third vendor; rung 4 of the ladder ("build on the SDK") is that architecture recommended to buyers rather than built by a vendor
• Harness Shrinkage as Models Improve — the thesis this puts under tension: prompts shrink, codebases and churn do not, and the pruning discipline is itself PR volume
• Agentic Loops Overtake Bespoke Systems — the same don't-build-bespoke conclusion reached by the capability route (scaffolding converts to a cost line as the model improves) rather than the maintenance route
• Agentic Technical Debt — the debt-family sibling with a different mechanism: intent drift inside your own code, not divergence from someone else's
• Agent Harness Engineering — what the ~1M lines is made of; this page prices the commitment that page describes building
• Agent Context Files — rung 1 of the ladder: prompts, tools, and defaults before code
• MCP and Computer Use — rung 2: integrations that survive an upgrade because they live outside the fork
• Agentic Work Systematization — rung 3: skills and plugins as org expertise without a permanent fork delta
• Verification as the New Bottleneck — the 866-bug-fixes-a-year figure is the maintenance half of that thesis measured on a harness rather than an application
• OpenHands — the source's author's employer and the subject of its most detailed numbers
• Codex — 7,688 merged PRs / ~1.32M lines / lowest bug-fix label share (16%)
• Hermes Agent — 7,736 merged PRs / ~1.75M lines / the 68% bug-fix outlier
• Claude Code — conspicuously absent from the comparison, being closed-source; the shrinkage claims it anchors are all prompt-side

Open Questions#

• What fraction of upstream churn does a narrow fork actually inherit? 13 PRs/day is the full-surface upper bound; nobody has measured the delta for a fork that drops the UI, multi-provider support, and deployment surface. A rerun restricting the diff to a realistic kept-file set would settle it.
• Does the merged-PR rate of coding-agent harnesses peak and fall as models improve, or keep rising? Harness Shrinkage as Models Improve implies an eventual peak; the observed trajectory (Codex ~124 → ~1,000 PRs/month over 12 months) is still climbing. Re-pull the same four repos in twelve months.
• Is bug-fix share comparable across projects at all, or is the 16–68% spread purely label hygiene? Shah says compare directionally and flags Hermes' 68% himself; a uniform commit-message classifier rerun across the four repos would separate signal from convention.

Sources#

• The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI — Sayed Ali et al. (33 authors, all Writer, Inc.; arXiv 2607.06906, 2026-07-08, empirical, total vendor COI): §3.3 why harness savings compound across every model in a fleet, §6.2 and Table 4 the per-model 33–61% cost reductions, §7.2 the $210k→$120k fleet arithmetic at one million tasks/month, §7.1 the own-versus-rent conclusion. Table 2 is cell-collapsed and Table 7 row-shifted in the raw parse; neither is cited
• Coding Agents and Technical Debt — Rajiv Shah, OpenHands blog, 2026-07-28 (case-study, vendor-interested): twelve-month GitHub activity for four coding-agent codebases, the four-repo OpenHands split, the fork-delta counterfactual, the customization ladder, and the method notes. Data pulled 2026-07-08 via GitHub GraphQL/REST. Table figures reconciled against the prose: the four repo rows sum exactly to the headline 5,679 PRs and ~1.05M lines