Pilot-to-Production Gap

Sources#

Deploying AI from pilot to production: A practical blueprint for CIOs and technical leaders

Summary#

Deploying AI from pilot to production (Anthropic × Accenture, 2026-09-11, 38pp, vendor-claim) argues that the enterprise AI pilot is a structurally misleading instrument: it succeeds because of conditions that production will not reproduce, so its success is not evidence about the system anyone will actually run.

"That insulation makes pilots successful, but it also makes them an unreliable signal for how enterprise AI will perform in production at scale."

The claim is not that pilots are run badly. It is that the pilot's insulation is the thing being measured, and the fix is to move the production decisions before the pilot rather than after it — which reframes the pilot from a capability demonstration into a readiness exercise. This is the practitioner statement of the thesis Organizational Complements to AI makes from the economics side: the pilot artificially supplies the complements, so the complement gap is invisible until the complements are withdrawn.

The insulation catalog#

The document's most transferable contribution is the itemization — what specifically makes a pilot unrepresentative. Six conditions, stated as the reasons pilots succeed:

Pilot condition	What production supplies instead
Curated datasets, prepared for the exercise	The real data estate: distributed, inconsistently formatted, governed by rules not designed for AI, owned by teams on other timelines
Handpicked, AI-native engineering teams	Standard delivery teams, "not representative of the broader enterprise workforce"
Protected budgets, insulated from normal org dynamics	Steady-state operational economics
Narrow, defined scope with clear timelines	Fragmented enterprise systems and bidirectional integration
A high level of manual intervention	Whatever the system does unattended
Limited exposure to downstream stakeholders	IT, Legal, Finance, HR and Compliance, each with a different definition of "working"

The fifth is the one that generalizes furthest and is named explicitly as a design rule — "minimizing hidden manual intervention". A pilot that quietly depends on a human repairing inputs measures a hybrid system whose human half is not in the production budget. It is the enterprise-deployment instance of the failure class Failures That Look Like Success names: the demo reads as working, and the part that made it work is not in the artifact.

The prescription is to build the production conditions into the pilot design: define production success criteria upfront, test against representative enterprise conditions, minimize hidden manual intervention, validate delivery scalability beyond AI-native specialists, embed governance and observability early, and measure long-term operational economics rather than short-term model performance.

Front-load the decisions#

The structural argument is about ordering, not content — every decision deferred past the pilot is claimed to compound:

"Every decision deferred past the pilot creates engineering debt: parallel systems, integration patterns that never standardize, and a platform foundation that's always being renegotiated."

Its sharpest illustration is the infrastructure hedge: an organization that cannot settle build-versus-buy runs the pilot on a managed API while a platform team builds in-house "for when we need more control", and six months later maintains two systems with every new use case reopening the debate. The document's claim is that most infrastructure problems come from never committing, not from committing wrongly — which is the same optionality argument Standardize the Infrastructure, Not the Tools resolves the other way, by making the substrate the committed layer and leaving tool choice deliberately open.

The deployment blueprint#

Seven decisions across four lifecycle stages, each with a named owner. Four of the seven land before the pilot begins, and the pilot stage itself introduces no new owners or decisions — it only tests the decisions already made.

Stage	#	Decision	Scope	Owner
Pre-pilot — decide before the pilot begins	01	Strategy and ownership	Success criteria, ROI thresholds, go/no-go gates	Executive sponsor
	02	Data and integration	Audit the data estate, assign source and pipeline owners	Data and IT change owners
	03	Infrastructure	Align platform choices to use-case requirements	CIO / platform lead
	04	Security and trust	Classify data; clear security, regulatory and access requirements	Security and compliance lead
Pilot — validate, don't defer	—	No new owners or decisions	The pilot tests the decisions already made	—
Preproduction — prepare the organization	05	Org readiness	Sequence rollout, fund reskilling, name champions	Exec sponsor / change lead
Production — govern by risk	06	Governance and risk	Risk taxonomy, monitoring thresholds, review authority	Risk and monitoring owner
Scaled deployment — operate as a capability	07	Scale and evolution	Start simple, scale deliberately; understand failure before scaling	AI program owner

(Source PDF p.35. This page renders as page graphics, so both docling and pdftotext recover nothing from it — the table above is a manual transcription made at ingest from the rendered page; see the parse note in Sources.)

Each consideration closes with a two-column question set the document splits by decision type: "Work out" questions need cross-functional input and have multiple owners; "Assign" questions are ownership decisions a CIO or business leader must make alone. The split is the document's operational core — it is a claim about which deployment questions are deliberative and which are executive, and treating an Assign question as a Work-out question is how ownership diffuses.

Ownership decomposes into three non-substitutable parts#

The named owner needs decision rights (make calls without convening a meeting), escalation authority (blockers reach senior attention in days, not months), and executive backing (signals institutional weight, which determines whether other functions engage as partners or observers).

"Decision rights, escalation authority, and executive backing aren't interchangeable, and the absence of any one of them is enough to stall a program."

The failure mode is measured, after a fashion: Accenture's September 2026 Tokenomics research reports that 42% of organizations rely on shared IT and finance accountability with no single owner responsible for AI costs and outcomes. Contrast AI Employee Framing, which finds the accountability leak coming from the framing of the agent rather than from the org chart (−9pp personal accountability, +44% escalation when an agent is called an "employee") — two independent routes to the same unowned outcome, one structural and one linguistic.

Deployment is not adoption#

The document separates shipping the system from the workforce using it, and is blunt that leadership pressure alone does not close the gap:

"A portfolio manager showing a compliance specialist how she summarized a 200-page filing in three minutes converts more skeptics than any structured rollout."

The prescription is to engineer the demonstration moments deliberately rather than wait for them — identify champions, clear time to experiment, recognize early adopters publicly — which is the same mechanism Standardize the Infrastructure, Not the Tools reports from Shopify (adoption by demonstration, not mandate), arrived at independently.

As adoption spreads, the claim is that work shifts from executing tasks to overseeing a system: identifying incorrect outputs, deciding what escalates, and detecting drift before it becomes a production issue. That shift is asserted, not measured, and it sits directly against the mechanism The Tragedy of the Cognitive Commons describes — if the entry-level execution work is what regenerates the judgment the oversight role requires, an org that converts everyone to overseers has removed the training path for the skill it now depends on. The document does not raise this.

The copilot throughput ceiling#

The scale-stage argument for end-to-end automation over assistance is a clean statement of where assistive AI stops paying:

"The throughput ceiling on a copilot is still the person using it. A tool that helps an analyst write reports faster is still bounded by how many reports that analyst can review and approve."

Only when the AI completes the work — invoices processed end-to-end with exceptions escalated, tier-one tickets from open to resolution — does "throughput scale with volume, not headcount". This is the enterprise-workflow statement of the shift Conversation-to-Delegation Shift measures in developer tooling (99.8% / 63.3% / 16.5% delegated-output share across three populations), and the tension it inherits is the same one Verification as the New Bottleneck names: removing the human from the loop moves the bound from production to verification rather than removing it.

Paired with a counterweight the document states plainly — start simple. A well-designed prompt is fast to test with predictable failure modes; a multi-step agentic system is more capable but harder to debug, more expensive to maintain, and fails in less anticipable ways. Add complexity only when the simpler approach has demonstrably hit its limits. (The document cites Anthropic's own Building effective agents for the same argument.)

And a readiness test that is genuinely sharper than an accuracy number: know the error mode distribution, not just the error rate. "A system that's 95% accurate at limited volume sounds production-ready. But the remaining 5% matters" — a formatting slip a reviewer catches in seconds may be safe to automate; an incorrect financial calculation or a missed compliance flag makes every additional unit of volume add risk. The operative question is what each type of error costs at full production volume. That is the consequence-tiering discipline AI-Assisted Error Analysis applies to eval investment, applied instead to the automation decision.

The figures, and how far they carry#

Every quantity here is vendor-published. The surveys are at least named and dated; the deployment anecdotes are not verifiable at all.

Attributed surveys (Accenture, self-published, methodology not included):

Figure	Source
23% of C-suite leaders report sustained, enterprise-wide AI impact	Pulse of Change, July 2026
64% have moved past pilots into production across multiple functions or begun enterprise-wide efforts — but only 7% have the data readiness to scale advanced AI	AI-Ready Data for Advanced AI, May 2026
"Data reinventors" realize EBIT margin uplift of up to 1.6× over industry peers	AI-Ready Data for Advanced AI, May 2026
42% rely on shared IT/finance accountability with no single AI cost-and-outcome owner	Tokenomics, September 2026
Organizations with formal chargeback accountability link 32¢ of every dollar of AI token spend to a quantified business outcome — 6× those with no allocation	Tokenomics, September 2026

Third-party: Gartner predicted (February 2025) that organizations would abandon 60% of AI projects unsupported by AI-ready data through 2026 — a prediction whose window closes this year and which the document cites without checking against outcome.

The 64% vs 7% pair is the load-bearing one, because it is the only figure that separates being in production from being able to scale, and it is the arithmetic behind the whole document. It should be quoted with the caveat that both halves come from one vendor's own survey of its own prospective buyers, and that "data readiness" is a threshold that vendor defines.

Unattributed anecdotes — no company, no methodology, no measurement protocol: a global insurer compressing underwriting review "more than 5×" while improving data accuracy "75 to 90%"; a pharmaceutical company cutting clinical study report production "from ten weeks to under ten minutes"; a telecom deploying "more than 13,000 custom AI solutions" in under a year. The named customer cases are thinner still as evidence, being Anthropic case-study links with a quoted executive: StubHub (30% support-cost reduction, response times from over 20 minutes to near-instant, after side-by-side model A/B testing); Novo Nordisk (the ten-weeks-to-ten-minutes claim, attributed); TELUS (Fuel iX multi-model platform, 13,000 custom solutions, 500,000+ hours saved, 100 billion tokens monthly, "Claude became the overwhelming choice"); Palo Alto Networks (20–30% increase in feature development velocity); NBIM (600+ active users within two months, an AI Ambassador Network of 50 specialists, internal surveys showing 20%+ weekly time saved).

Read these as existence proofs, not effect sizes. The selection is the vendor's, the counterfactual is absent everywhere, and in the two cases where the same number appears both as an anonymous statistic on p.3 and as a named case later (Novo Nordisk, TELUS), the anonymous framing implies a breadth of evidence the named cases do not supply.

What this document does not do#

Three things a reader should not expect from it:

No failure analysis. Every case is a success; the ~77% of programs the document's own headline statistic says have not reached sustained impact are characterized only by which considerations they skipped, which is an assertion of the thesis rather than evidence for it.
No priors on the decisions themselves. The blueprint says who decides and when, never what to decide. Build-versus-buy, single-versus-multi-model, cloud-versus-on-prem are all resolved into "commit deliberately" — which makes the document a process artifact, not a technical one. The one exception is the retrieval-architecture note below.
No cost of the process. Front-loading seven decisions before a pilot has a price in calendar time and executive attention, and it is not estimated anywhere — which matters, because the prescription's own premise is that "AI is compressing that timeline into months or even weeks".

The one architectural claim#

Consideration 03 carries the document's only substantive technical recommendation, and it is a subtractive one: audit whether the retrieval layer is solving a current problem or an expired one.

"Many existing pipelines are solving for a context window constraint that no longer applies."

Chunking, pre-send summarization and staged retrieval were workarounds for small context windows; modern models process whole contracts, codebases or research reports in one pass. The document allows that retrieval still earns its place where data freshness or access control are the drivers — which is the same residual Document Parsing as the Retrieval Bottleneck identifies from the opposite direction (long context did not kill RAG; cost, governance and audit keep it), and the same expiry logic Context Window Smart Zone complicates, since usable context is measured well below the advertised window.

Connections#

Organizational Complements to AI — the economics of the same claim. That page's thesis is that AI productivity gains depend on complementary workflow, skill and org-design changes; this document is the practitioner-side account of why the dependency stays hidden, since a pilot supplies the complements artificially (curated data, AI-native staff, protected budget) and therefore cannot measure the gap it will hit. The 64%-past-pilots-vs-7%-data-ready split is a complement gap stated as a survey statistic — with the caveat that it is one vendor's survey, where that page's core evidence (the Codex three-population natural experiment) is not
Risk-Tiered Auto-Approval — consideration 06 is a fourth tiering key, keyed to the consequence of the output rather than the change, the anomaly, or the reversibility of the response: a four-tier oversight ladder (automated / sampled / reviewed / advisory) with review cadences attached, plus the checkpoint-audit discipline and the "structural drag" account of why review layers accumulate. Detailed on that page
Standardize the Infrastructure, Not the Tools — the same cost-governance layer, with the chargeback numbers this document supplies (32¢ of every AI-token dollar tied to an outcome under formal chargeback, 6× those with none; 42% with no single owner) and the same adoption-by-demonstration mechanism. But the two resolve the commitment question oppositely: this document treats an uncommitted build-versus-buy as the primary source of engineering debt, where Shopify deliberately buys optionality by standardizing only the substrate
Failures That Look Like Success — "minimizing hidden manual intervention" is this failure class located in the deployment pipeline rather than in an agent trajectory: the pilot reads as working, and the human repair that made it work is neither in the artifact nor in the production budget
Conversation-to-Delegation Shift — the copilot throughput ceiling is this shift argued prospectively for enterprise workflows, where that page measures it retrospectively in developer tooling. The asymmetry is worth noting: this document asserts that throughput scales with volume once the human leaves the loop and supplies no instance where it was measured, while that page has the delegated-output shares and no claim about the ceiling
AI Employee Framing — the second route to unowned AI outcomes. This document locates the accountability leak in org structure (42% with no single owner) and prescribes a named owner with three non-substitutable powers; Kropp et al. locate it in language, and find that calling the agent an "employee" costs −9pp of personal accountability with no adoption gain. Both failures survive the other's fix
The Tragedy of the Cognitive Commons — the unraised tension. The document's consideration-05 prescription is that work shifts from executing tasks to overseeing a system, requiring "judgment-based skills"; Lovett's argument is that those skills regenerate through exactly the entry-level execution work being automated away. The document treats the oversight capability as a training investment; that page treats it as a commons with a removed regeneration mechanism. Neither cites the other, and this is the sharpest thing the vault can say against the blueprint
Document Parsing as the Retrieval Bottleneck — the same verdict on retrieval's remaining job from the opposite approach: this document argues subtractively (audit whether the pipeline solves an expired context-window constraint), that page argues from the 2024→2026 retrospective that long context did not kill RAG because cost, governance and audit outlive the constraint. They agree on the residual — freshness and access control — and disagree on nothing
Verification as the New Bottleneck — where the end-to-end automation argument runs out. The document's error-mode-distribution test (what does each type of error cost at full volume, not what is the error rate) is a deployment-side statement of that page's problem, and its escalate-only-the-exceptions design assumes the exception detector is the cheap part
AI-Assisted Error Analysis — the same consequence-first reasoning applied to a different decision: that page sets eval investment by reasoning through worst-case user outcomes up front, this one sets the automation boundary the same way. Both reject a uniform bar across features
Anthropic — publisher, with Accenture. The document's "Getting started" section is a reading list of Anthropic product and engineering material (Claude Enterprise Administrator Guide, Scaling agentic coding across your organization, Claude Cowork enterprise controls, Building effective agents, Effective context engineering, MCP), which is the clearest single indicator of what the artifact is for
Claude Code — named as the worked example for the leading-vs-lagging indicator argument: PR cycle time is a leading indicator of value from a developer using Claude Code, but business value is the lagging indicator (revenue, net revenue retention, churn) and must be attributed to what shipped

Open Questions#

Gartner's February 2025 prediction — organizations abandon 60% of AI projects unsupported by AI-ready data through 2026 — reaches its window this year, and this document cites it without checking it. Did the abandonment rate land near 60%, and was data readiness the separating variable it was predicted to be? A resolved forecast would convert the document's central data-readiness claim from a vendor survey into something gradeable.
The blueprint's load-bearing claim is an ordering claim: decisions made pre-pilot cost less than the same decisions made post-pilot. Nothing in the document measures this, and the counterfactual is available in principle — two programs, same use case, differing only in whether ownership, data audit, infrastructure and security were settled before the pilot. Does front-loading actually shorten time-to-production, or does it move the same calendar time earlier and add executive-attention cost the document never prices?
Consideration 05 prescribes converting operators into overseers ("the claims processor who used to review every invoice now audits the system that reviews them") while The Tragedy of the Cognitive Commons argues the judgment that role needs is regenerated by the execution work being removed. Is there a deployment where an oversight workforce was staffed without an execution cohort beneath it, and did its error-catching hold over more than a year? This is the falsifiable core of the disagreement and neither source supplies it.

Sources#

Deploying AI from pilot to production: A practical blueprint for CIOs and technical leaders — Deploying AI from pilot to production: A practical blueprint for CIOs and technical leaders, Anthropic × Accenture, 2026-09-11, 38pp, vendor-claim. Co-branded enterprise collateral: the authors sell the product the guide recommends, the "Getting started" section is a list of Claude product links, and every deployment figure is either an unattributed anecdote or a vendor case-study page. The prescriptive content — the insulation catalog, the Work-out/Assign split, the three components of ownership, the four-tier oversight ladder, the error-mode-distribution test — is practitioner judgment and is where the document's value sits; the numbers are Accenture's own surveys (Pulse of Change July 2026, AI-Ready Data for Advanced AI May 2026, Tokenomics September 2026), self-published without methodology. Parse warning: p.35's deployment blueprint is drawn as page graphics — docling recovered only its decorative timeline dots and pdftotext recovers only the running head, so the page reads as present while carrying none of its content. The table on this page is a manual transcription made at ingest from the rendered page and is the only copy of that content in the vault; the source's own typo ("No new owners or decsions") is preserved in the raw with [sic]. The nine extracted tables parsed clean