Why AI pilots don't reach production

Most enterprises now have several successful AI pilots and very few production agents. That gap is not a technology problem, and it is rarely a model quality problem. It is a set of five recurring, predictable blockers — and because they are predictable, they can be designed out before the pilot starts rather than discovered after it succeeds.

This is what we see across delivery engagements, and what actually changes the odds.

1. The pilot excluded the hard part

Pilots are usually scoped to the happy path, because the happy path demonstrates well. But the happy path is not where the cost is. The cost is in the exception queue — the cases the current rules do not cover, which is precisely why humans still handle them.

An agent that handles the 70% already handled cheaply produces a modest saving and an unconvincing business case. The same agent aimed at the 30% that consumes most of the team's week produces a very different number. Scope the pilot at the exceptions, even though it demos worse, because that is the version anyone will fund.

2. Nobody owns it

A pilot belongs to whoever built it. A production agent needs a named owner who is accountable when it does something wrong at 2am — and that person has to exist before go-live, not be identified during the first incident.

This sounds like process rather than engineering, and it is the single most common reason a working agent sits in staging for a quarter. Ownership needs to be a recorded property of the agent — owner, version, applicable policy — not a convention someone wrote in a wiki during the project.

3. The risk review has nothing to review

The review board asks who approved a given action, what the agent was looking at when it decided, and what happens when it gets one wrong. If the answers are architectural intentions rather than artefacts, the review does not fail — it adjourns. Then it adjourns again.

The fix is to bring the artefacts to the first meeting: the policy pack showing which tools the agent may call and which actions need a named approver, and a replayable run showing those controls firing. Reviews that start with evidence tend to finish.

4. The demo used data the agent will not have

Pilots frequently run against an extract, a copy, or a curated document set, under a service account with broad permissions. Production runs against live systems, under a real user's entitlements, with rows that are missing fields and documents that are scanned sideways.

The accuracy measured in the pilot does not transfer, and the difference is discovered at the worst possible moment. Run the pilot under the permissions the agent will actually have, against the systems it will actually query, or accept that the number you produced is not evidence of anything.

5. Going live means rebuilding it

If the prototype was built on one stack and production requires another — different integration layer, guardrails re-implemented, a front end built separately — then "go to production" is a second project with its own budget cycle. Many pilots die here, not because they failed but because nobody funded the rebuild.

The question to ask before the pilot starts is simple: what exactly changes between this working and this being live? If the answer is a list, that list is your real timeline.

What changes the odds

The programmes that reach production tend to share a shape, and it is not a more capable model.

• Map the process as it is actually performed, exceptions included, rather than as it is documented.
• Encode the guardrails before building the agent, so the constraints shape it rather than being bolted on.
• Run in shadow mode against real work, proposing but not executing, and measure against what actually happened.
• Promote when the measured numbers justify it, in one controlled action rather than a re-implementation.
• Name the owner before go-live, and record it in the platform.

Shadow mode is the step most programmes skip and the one that most reliably gets them through the review, because it converts an argument about whether the agent is good enough into a measurement.

Where this advice does not apply

Two honest caveats. First, some processes should not be automated at all — where volume is low, the judgement is genuinely novel each time, or the cost of a rare error exceeds the total saving. A good discovery phase produces some of these answers, and saying so early is more valuable than a pilot that quietly underperforms.

Second, timelines vary. A first governed agent in around six weeks is realistic when the process is understood, the systems have APIs and the approvers are identified. It is not realistic if the process is undocumented, the data needs remediation first, or the organisation has not decided who owns the outcome. The constraint is usually organisational readiness, not engineering.