Client-Side Agent Optimization

Sources#

• Agent swarms and the new model economics
• AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agent
• Recursive Self Improvement for Coding Agents
• The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI
• Tips & Best Practices

Summary#

A framing introduced by Hua et al. (AgentOpt, 2026) that separates client-side optimization of agentic workflows — decisions under the developer's control such as which model to assign to each pipeline role, API budget allocation, and tool routing — from the server-side techniques (caching, scheduling, speculative execution, load balancing) that have dominated systems research on LLM serving. The central empirical claim is that model selection, evaluated at the level of full pipeline combinations rather than per-role in isolation, is the dominant efficiency lever: cost gaps between best and worst combinations at matched accuracy range from 13× to 32× across benchmarks, dwarfing what server-side optimizations can recover.

Details#

Server-Side vs. Client-Side#

Server-side systems (vLLM, SGLang, Autellix, ThunderAgent, Continuum, AIOS) optimize provider infrastructure across many users with objectives like throughput, tail latency, and cluster utilization. These objectives are generic because the provider cannot see the developer's specific utility function. Client-side optimization operates at the level of a specific workflow with an application-specific utility over quality, cost, and latency — a startup's coding assistant and a clinical-support system have incompatible preferences that can't be inferred from system-level signals.

The resources under client control:

• Foundation model pool — available API and local models
• Model-to-role assignment across planners, solvers, critics, retrievers
• Tool invocation policy — local vs. remote, when to skip
• API budget per step
• Application-level batching, caching, scheduling

Why Model Selection Is First-Class#

Model selection is upstream of every other client-side optimization: caching, routing heuristics, and speculative execution all operate conditional on a model assignment. Pick the wrong combination and no downstream optimization can close the gap.

The empirical evidence is striking. On BFCL, Qwen3 Next 80B matches Claude Opus 4.6 in accuracy at 32× lower cost. On MathQA, 24× gaps exist between comparably-accurate combinations.

The Combo Abstraction#

The paper's key conceptual contribution. In conventional LLM routing, each query is assigned to a cheaper or stronger model based on estimated difficulty — decisions are per-call. In multi-step agents, routing decisions are coupled across stages: a model's behavior in one role changes the intermediate state that later roles see. A planner that delegates to a tool creates different downstream work than a planner that answers from parametric knowledge.

Consequence: the unit of optimization is the full combination $\mathbf{c} = (m_1, \dots, m_H) \in \mathcal{M}^H$, not the per-role best. Performance rankings do not transfer cleanly across roles — a strong standalone model can be an excellent solver but a poor planner.

The canonical illustration from the paper, HotpotQA:

• Claude Opus 4.6 is the worst planner across 81 combinations — when used as planner it often answers directly from parametric knowledge and bypasses the solver's search tools.
• Ministral 3 8B is the best planner because it reliably delegates to the downstream solver.
• Ministral (planner) + Opus (solver) → 74.27%; Opus (planner) + Opus (solver) → 31.71%.

This is the same overthinking / overelaboration phenomenon described in Scale-Dependent Prompt Sensitivity, surfaced as a routing failure rather than a prompt-engineering failure.

Formulation as Black-Box Optimization#

Given pipeline roles $H$ and candidate set $M$, the combination space is $|M|^H$ — exponential. The utility function

$$J(\mathbf{c}) = \mathrm{PERF}(\tau(\mathbf{c})) - \lambda_c,\mathrm{COST}(\tau(\mathbf{c})) - \lambda_\ell,\mathrm{LATENCY}(\tau(\mathbf{c}))$$

is treated as an unknown black-box because cross-stage interactions are task-dependent and not analytically tractable.

Search Algorithms#

AgentOpt implements eight selectors sharing the same execution substrate:

• Arm Elimination (best-performing) — multi-armed bandit that prunes dominated combinations. Recovers near-optimal accuracy at 24–67% less evaluation budget vs. brute force on 3/4 benchmarks.
• Epsilon-LUCB — confidence-bound bandit
• Threshold Successive Elimination
• Bayesian Optimization
• Plus hill climbing, random search, and brute-force baselines

All selectors share the same API so strategies can be swapped without touching agent code.

Framework-Agnostic Interception#

The systems mechanism: patch httpx.Client.send and httpx.AsyncClient.send at the HTTP transport layer. Attribution of each call to (datapoint, combination) uses Python contextvars. This avoids per-framework SDK adapters — works across Langgraph, AutoGen, OpenClaw, Claude Code, and any agent using httpx under the hood.

The runtime also handles response caching (re-runs of the same (combo, datapoint) pair don't re-spend the API budget) and parallel execution (e.g., max_concurrent=20).

Output: a SelectionResults object exposing the Pareto frontier over (performance, cost, latency), with CSV export and YAML configuration export for deployment.

Separation of Policy and Execution#

Selectors (what to evaluate next) are separate from the runtime (how to execute, track, attribute, cache). This separation is what lets the eight algorithms share benchmarks — the search is the only variable.

Manual Levers in the Wild#

The client-side levers AgentOpt formalizes (model assignment, budget, caching, batching) appear as user-facing CLI commands in production agent tools. Hermes Agent is the most explicit:

Hermes lever	AgentOpt analog
/model (mid-session model switch)	per-role model assignment in the combo space
/compress (summarize conversation)	application-level caching / context-budget management
/usage, /insights	observability over the same cost/latency/perf signals AgentOpt uses for utility
delegate_task (parallel subagents with isolated contexts)	sub-pipeline assignment with independent combos
Bounded MEMORY.md (~2,200 chars), USER.md (~1,375 chars)	explicit budget envelope on persistent context
Prompt-cache discipline (avoid mid-session model/system-prompt changes)	the cache-stability constraint that makes per-session combo selection stable

Significance: the levers exist in production tools today and are exercised manually by users. AgentOpt's contribution is automating selection over the same lever space rather than introducing new levers. A practical bridge would be an AgentOpt selector that drives Hermes's /model switches per-role given a benchmark, then writes the resulting combo into AGENTS.md for deployment.

The Hermes documentation also captures a constraint AgentOpt's combo abstraction implicitly relies on: don't break the prompt cache mid-session. Cache hits make per-message cost roughly constant; mid-session model/system-prompt changes invalidate that. If combo selection changes per-call rather than per-session, expected savings can be wiped out by cache misses — a deployment hazard worth surfacing when promoting AgentOpt's findings to production.

The combo abstraction, run in production (Cursor, 2026)#

AgentOpt's 13–32× cost gaps are benchmark measurements over synthetic pipelines. Cursor's swarm post (Agent swarms and the new model economics, 2026-07-20, case-study) is the same abstraction on a four-hour production build — a two-role pipeline (planner, worker), four combinations, matched task and matched time budget, with the outcome published in dollars (Cost-per-Task Over Cost-per-Token, Parallel Agent Orchestration).

Three things it adds that the benchmark could not:

• The gap survives at production scale and stays cost-shaped. All four combinations landed at comparable quality; total cost spanned $1,339 to $10,565, and worker spend alone spanned $411 to $9,373. AgentOpt's central claim — that equally-accurate combinations differ enormously in cost — reproduces on a real workload without a bandit anywhere in the loop.
• Coupling shows up in cost, not just quality. The combo abstraction exists because "a model's behavior in one role changes the intermediate state that later roles see." Cursor supplies the monetary form: the Fable 5 planner cost slightly less than the Opus 4.8 planner (fewer planning tokens despite ~2× the price) while its workers burned several times as many tokens, making the run substantially more expensive overall. A role's cost is not separable from the combination, so per-role optimization can pick the locally cheaper model and lose.
• The "worst planner" result is a harness property, not a model property. AgentOpt found Opus 4.6 the worst planner across 81 HotpotQA combinations because it answered from parametric knowledge instead of delegating to the solver's tools. Cursor's design forecloses that move by construction — "planner agents split a goal into pieces and delegate them… a planner never implements" — and with the option removed, frontier models in the planner seat are the cheap configuration. Read together, the AgentOpt result is best stated as: a planner that can execute will, and strong models are the most likely to. The fix is available in two places, and Cursor took the architectural one.

Weight it as vendor case-study, on Cursor's own harness, with Cursor's own Composer 2.5 as the worker in both hybrid arms — and note that the four mixes were never crossed into the full N×N matrix, which Cursor names as future work. There is no ablation and no search; this is one hand-picked point per corner, not a frontier.

A third routing axis: feature demand#

AgentOpt routes on role; conventional routing routes on estimated difficulty. Writer's harness-swap paper (The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI, empirical, total vendor COI — see Orchestration Sets Token Economics) argues for a third axis, and supplies the measurement that motivates it.

Holding six models fixed and swapping only the orchestration layer, efficiency gains were model-invariant (every model 33–61% cheaper) while quality gains scaled almost perfectly with baseline model strength (r = 0.99, and the weakest model was net-negative). All seven regressions across 48 capability × model cells landed on the three smaller models, concentrated in orchestration-heavy capabilities — MCP tool use (Qwen −0.15, GLM −0.06, Flash −0.04), Playbooks, Presentations — while the frontier models improved most in exactly those categories. Sub-agent delegation, the harness's one net-new feature, is only usable on the two strongest models (0.85–0.86 versus 0.42–0.45 on the fast tier).

The consequence for combo selection: an orchestration feature carries a capability floor, so a request should be routed by the features it will exercise, not only by how hard its text looks. A request that will spawn sub-agents belongs on a strong model regardless of apparent simplicity; a grounded Q&A request can take the 61%-cheaper fast tier with no quality penalty (grounding improved for every model in the panel). This is a cheaper heuristic than combo-level search over the same space, and it is orthogonal to the planner/worker positional rule that Cursor's mixes establish — the two compose.

It also relocates part of the combo space. AgentOpt treats the harness as fixed and searches over model assignment; this measures the harness as the larger cost term (33–61% per model versus 36% across the entire model menu on that workload), which would make orchestration configuration a lever sitting above the one AgentOpt optimizes. Weight the magnitudes as vendor-specific — one baseline loop and one harness, both from the company publishing the result.

Connections#

• Orchestration Sets Token Economics — the layer AgentOpt holds fixed, measured: swapping only the orchestration code moved cost per task more than the entire spread of the model menu did. It also sharpens routing with a feature-demand axis (orchestration features carry capability floors) and states the complement cleanly — "routing chooses which model pays the bill; the harness determines how large the bill is for whichever model is chosen"
• Prompt-Cache Economics — the same hazard given a cost model and a measured failure. Two things bear directly on this page: the cost-aware routing lineage it situates itself against (FrugalGPT, RouteLLM, xRouter, Cascade Routing) treats per-model price as a static constant, and CAPC shows it is a function of cache state, prefix size, and call count — composing ρ(N,|P|) with a router is named as open work. And the Hermes cache-discipline row below understates the problem: below Anthropic's ~3,500-token tier boundary a byte-identical prefix still misses ~17% of the time, and on a small-prefix agent workload explicit cache_control bought +0.6%, i.e. nothing
• Context Lifecycle Management — the "don't break the prompt cache mid-session" hazard turned into a commit-time decision rule with a published threshold (commit a context edit only once expected pruning exceeds 0.3, else hold the plan until cache expiry)
• Evals as Product Spec — good evals are what make per-role model optimization measurable
• The Verifiability Thesis — the A/B/C/D cost-vs-solve frontier optimizes within verifiable rewards
• Scale-Dependent Prompt Sensitivity — AgentOpt's HotpotQA finding (Opus is the worst planner because it bypasses the solver) is the same overthinking / over-elaboration mechanism Hakim documents at the prompt level. One paper surfaces it as a routing failure, the other as a prompt-engineering failure; together they imply that large-model misuse is a systematic failure mode with two available mitigations (route around it, or constrain output)
• Agent Harness Engineering — client-side optimization is a layer above harness design: once the environment, progress logs, and verification loops are in place, combo selection chooses which models operate inside that harness. The JSON feature-list and progressive-disclosure patterns are execution substrate for the agents AgentOpt assigns
• Claude Code Best Practices — directly challenges the implicit "use the strongest model" default. AgentOpt's framework-agnostic httpx interception is also compatible with Claude Code's claude -p non-interactive mode, suggesting Claude Code pipelines can be subject to combo optimization
• LLM-Driven Vulnerability Research — the file-ranking 1–5 pre-pass and the final validation agent are hand-tuned instances of exactly what AgentOpt searches over automatically. Treating the vuln-research scaffold as an AgentOpt pipeline (planner = file-ranker, solver = bug-finder, critic = validator) is a direct generalization
• LLM-as-Compiler Knowledge Base — the wiki's own compile / query / lint phases could be modeled as an agent pipeline where different phases run on different models (e.g., cheap model for index drift checks, strong model for cross-reference synthesis)
• Claude Opus 4.7 — the HotpotQA planner failure was measured on Opus 4.6; 4.7's literal instruction following may partially close that gap (needs re-measurement). Task budgets (public beta) echo AgentOpt's budget lever, but server-side rather than client-side
• Claude Sonnet 5 — a vendor-shipped instance of the same lever: Anthropic pitches dialing the effort parameter to slide Sonnet 5 along a cost-performance curve that overlaps Opus 4.8, so model-vs-effort choice is the client-side budget decision, now surfaced as a first-party product knob
• Hermes Agent — production CLI agent that exposes the AgentOpt lever space (/model, /compress, delegate_task, bounded memory, prompt-cache discipline) as user-facing commands; a natural integration target for AgentOpt selectors driving role assignment automatically
• Symphony — at scale, ticket-driven orchestration makes per-pipeline combo selection operationally important: choosing the right model per ticket type (planner vs. solver vs. reviewer) inside WORKFLOW.md's prompt template is a per-pipeline budget decision
• Codex App Server Protocol — agent.max_turns, turn/stall timeouts, and dynamic-tool-call cost are operational instances of the budget lever AgentOpt formalizes
• Interaction / Background Model Split — another axis of multi-model design: there cost-driven and static per role, here latency-driven and dynamic per turn
• Ticket-Driven Agent Orchestration — at orchestration scale, choosing the right model per ticket type (planner/solver/reviewer) inside WORKFLOW.md is a per-pipeline instance of combo selection
• Evolutionary Proof Search — model-per-role made concrete: DeepMind runs Gemini 3.1 Pro for proving and the cheaper 3.0 Flash for rating — an explicit cost/quality combo inside one agent
• AI-Driven Formal Proof Search — the A/B/C/D solve-rate-vs-cost Pareto curves are the same cost/quality optimization AgentOpt formalizes; here the cheaper config often wins (Agentic Loops Overtake Bespoke Systems)
• Deep Research Agents — DRACO's token/latency table is this cost/quality framing in the deep-research setting: more output ≠ better, and orchestration beats raw token spend
• Cost-per-Task Over Cost-per-Token — the vendor-side counterpart, and a direct tension: Anthropic tells developers to start with the strongest model and dial effort down, while AgentOpt's HotpotQA result shows the strongest model as the worst planner. Weight by tier — AgentOpt is empirical and multi-role, the guidance is vendor-claim and single-role — and note the guidance's own hedge (Sonnet for high-volume sub-agents). Its advisor strategy (cheap worker calls a strong advisor to check plan and work) is a first-party client-side lever with a number attached: Sonnet 5 + Fable 5 advisor lands within 10% of Fable 5 on SWE-bench Pro at 63% of the price
• Repository Exploration Subagent — extends model-per-role one step further: train a bespoke small model for the explorer role rather than assigning an off-the-shelf one; FastContext's 4B-RL explorer beating a 30B-SFT one is combo optimization where the winning "model" is fine-tuned, not picked
• Agent-Authored Harness Optimization — the sibling lever, searched by an agent rather than a bandit: AgentOpt optimizes model assignment with the harness fixed; Cline's campaign optimizes harness patches with the model fixed. Both are client-side, and no work in this corpus measures them against each other
• Parallel Agent Orchestration — the harness Cursor's combinations ran inside; the coordination rebuild is the other half of that experiment, and the reason the four mixes are comparable at all
• Cursor — the vendor, its own model in two of the four arms, and how to weight the figures
• Orchestration-Plan Simulation — the HotpotQA planner result corroborated on a second, independent instrument. OrchBench isolates the planner role by construction (workers are simulated, so only the plan varies) and finds no model leads at every scale — GPT-5.5 at 10 subtasks, GLM-5.1 at 20, Claude-Opus-4.8 at 50, Gemini-3.1-Pro at 100, with top-two gaps as small as 0.0007 within a source family — plus distinct per-model orchestration styles: Claude-Opus-4.8 conservative on handoffs (near-zero missing transfers, quality-preserving, mediocre on the speed/token half of the composite), Doubao over-launching agents while under-declaring transfers. Read together with AgentOpt's "strongest model is the worst planner," planner ability is not a projection of general model strength, now measured twice by different methods. It also reframes the combo space: OrchBench's ablation says essentially all planner separation comes from information preservation, so the role a router most needs to get right is the one that decides what gets passed forward

Derived#

• When to Use Claude Opus 4.6 for Work — deployment rules drawn from the HotpotQA planner/solver results and the BFCL 32× cost-match finding
• Opus 4.6 → 4.7 Changes and Multi-Agent Coding Considerations — role-based model selection principles applied to an Opus 4.7 multi-agent coding team

Open Questions#

• How does combination-level optimization interact with continual model releases? If Claude Opus 4.7 ships next month, does the full Pareto frontier need re-running, or do warm-started bandits adapt cheaply? Partially answered (2026-08-04, by synthesis): What Makes a Self-Improvement Artifact Transfer? — a combination is a solver-fitted artifact (fitted to the current menu's capabilities and prices), so the frame predicts full re-runs rather than cheap adaptation, with the durable residue being assignment rules rather than assignments; the HotpotQA→Cursor inversion below already shows the assignment not transferring while the reconciling rule does. The warm-started-bandit half is an empirical question no source measures.
• At what pipeline depth does the combinatorial search become intractable even for Arm Elimination? The paper tests up to ~81 combinations; production pipelines with 5+ roles and 10+ candidate models each blow past that.
• Does the "weak planner + strong solver" pattern generalize, or is it specific to HotpotQA's delegation dynamic? Recommender-critic, drafter-editor, and retriever-generator topologies might invert. Partially answered — it inverts (2026-08-03): on Cursor's long-horizon build task the efficient frontier is the opposite assignment, strong planner + cheap worker, with the entire worker fleet costing $411 under an Opus 4.8 planner versus $9,373 when a frontier model did both jobs at the same quality. The reconciling variable is what the planner is able to do: HotpotQA's planner could answer directly and did; Cursor's cannot. So the pattern is specific to the delegation dynamic, and the general rule is about foreclosing execution at the planner role rather than about weakening the model in it.
• What's the right way to re-evaluate when the tool environment changes? AgentOpt assumes fixed tools — adding or removing a tool potentially invalidates the whole frontier.
• Is there a cheap per-call classifier that can predict which combination will win on a given query, avoiding combo-level evaluation entirely? Sharpened (2026-08-03): Writer's harness swap proposes classifying on feature demand rather than difficulty — which orchestration features (delegation, MCP tool use, multi-step workflows) a request will exercise — on the evidence that those features carry capability floors and that a model below the floor fails on them regardless of how simple the prompt reads. That is a candidate classifier target, not a classifier: nobody has built or evaluated one.

Sources#

• AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agent
• The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI — Sayed Ali et al. (33 authors, all Writer, Inc.; arXiv 2607.06906, 2026-07-08, empirical, total vendor COI): §6.2–6.5 model-invariant efficiency, harness leverage (r = 0.99), the seven regressions and the sub-agent capability floor; §7.4 routing by feature demand; §2 the "routing chooses who pays, the harness sets how much" framing. Table 2 is cell-collapsed and Table 7 row-shifted in the raw parse; neither is cited
• Agent swarms and the new model economics — Wilson Lin, cursor.com, 2026-07-20 (case-study, vendor-authored): "Trees and leaves" (the planner-never-implements rule), "Results across model mixes" (the four planner/worker combinations), "Model economics" (the $411-versus-$9,373 worker spend and the Fable-versus-Opus planner inversion)