The Demo Worked. Six Months Later, It Is Still a Pilot

A pattern keeps repeating in enterprise AI: the proof of concept works.

A chatbot over internal documents. A copilot that summarises reports. An agent that clears a repetitive queue. The demo lands, the room nods, the budget gets approved.

Then the calendar moves and the project does not. Requirements shift. Security appears with a list. Costs get vague. The system that was crisp on curated files turns unreliable on the real thing. Nobody kills it. It just keeps getting called a pilot until people stop asking.

I have watched this from inside the engineering room more times than I would like. And the part that still surprises people: the pilots that stall are rarely the weak ones. They stall because the demo succeeded early enough to convince everyone the hard part was over.

One failure appears earlier than most teams expect: the pilot often starts without a written definition of what would qualify it for production. That becomes important later.

Short answer: A POC and a production AI system answer different questions. A POC tests whether the idea can produce useful output in a controlled environment. Production tests whether the organisation can operate that system reliably, safely, economically, and inside a real workflow. We assess that transition through six lenses: data reality, infrastructure, governance, operations, people and delivery, and goals. Stalled pilots usually expose gaps across more than one of them.

The number nobody quotes

You have seen the headline. MIT's Project NANDA report, The GenAI Divide: State of AI in Business 2025, found roughly 95% of GenAI pilots produced no measurable impact on the profit and loss statement. It got quoted into meaninglessness within a week.

The figure underneath it is the one that should worry you, and it is almost never cited. For enterprise-grade tools, about 60% of organisations evaluated them. Around 20% reached a pilot. About 5% reached production.

Read as a funnel, the important gap sits between pilot and production. If roughly 20% reach pilot and around 5% reach production, the pilot itself is clearly not proof that the organisation is ready to scale the system.

S&P Global Market Intelligence found the pattern accelerating rather than improving: the share of companies abandoning most of their AI initiatives climbed from 17% to 42% year over year, with the average organisation scrapping 46% of proofs of concept before production. Cost, data privacy and security topped the obstacle list.

And Gartner predicted over 40% of agentic AI projects would be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls.

Now a small thing that tells you how carelessly this whole field reads its own evidence. Gartner published that prediction in June 2025. Most of the coverage circulating today drops the date and presents it as a fresh 2026 finding. If a statistic in an AI article is undated and unlinked, it is decoration. The same discipline you are about to apply to your pilot applies to the research about pilots, and almost nobody applies either.

The demo was never the system

A POC is not a small production system. It is a different artifact with a different job, and it is completely honest about that job right up until someone mistakes it for a foundation.

It runs on a folder someone cleaned by hand. It serves the people who made it. It answers the questions they thought to ask. Everyone sees everything, because permissions were out of scope. Someone watches it during the demo, which counts as monitoring. Cost never comes up, because the volume is a rounding error.

Every one of those is a deliberate simplification, and a correct one. Each is also a system that does not exist yet.

The POC proves the idea is possible. Production proves your organisation can run it. Those are not the same claim, and only one of them was tested.

Your dashboard is green. Your system is broken.

One production lesson changes how you monitor AI systems: a healthy service does not necessarily mean a healthy AI system.

AI introduces a failure mode that conventional infrastructure monitoring is not designed to catch. The system can remain available, responsive, and error-free at the API layer while the quality of its output quietly degrades.

Traditional services usually fail in ways our operational tooling understands. A request returns a 500. A queue backs up. Latency spikes. An alert fires, someone gets paged, and the incident is visible.

AI systems can fail very differently.

They can return a fluent, confident, well-structured answer that is wrong, while still responding with a 200 status code in 800 milliseconds. Uptime looks healthy. Error rates stay flat. Latency remains within target.

From an infrastructure perspective, the system is working. From the user's perspective, it is not.

That is why production AI needs another layer of operational visibility alongside uptime and latency: evaluation, sampling, traceability, and quality monitoring that can detect when the system is still running but no longer behaving well.

Sit with that for a second, because the consequences are not small. It means nobody gets paged. It means quality can decay for weeks before a human finally escalates, by which point trust is gone and the pilot is already politically dead. It means the failure is invisible precisely to the people funding it.

You cannot alert on an answer that was merely wrong. You have to go looking for it, deliberately, on a schedule, with evaluation sets and sampling and traces. That is not a tool you install. It is a practice you staff.

Which is why, when a client asks us to review a stalled pilot, the first artifact we ask for is not the architecture diagram. It is the evaluation set. If it does not exist, we already know most of what we are going to find, and so does everyone in the room the moment the question is asked.

Six lenses between a POC and Production

We do not start with the model, vendor, or prompt. We look at six parts of the operating system around the pilot. A weakness in one can slow deployment. Several interacting weaknesses are what usually make a project feel permanently stuck.

Data reality: the person was the pipeline

A pilot usually starts with cleaner data than production ever gets.

Someone selected the documents. Someone removed duplicates. Someone knew which version of the policy was current. Someone corrected the file names, ignored the broken PDFs, and quietly filled in the context the system did not have.

That work often disappears from the architecture diagram because it happened manually.

Then the pilot moves toward production and the system meets the real data estate: scanned documents, inconsistent formats, stale versions, missing metadata, duplicated records, access restrictions, and source systems that disagree with each other.

The AI did not suddenly become less capable. The human preparation around the pilot was doing more work than the team realized.

Production has to turn that invisible preparation into a repeatable process: ingestion, parsing, versioning, metadata, access rules, quality checks, and clear ownership of the source data.

This is why data readiness is not simply about having enough documents to index. The system needs to know which information is current, which source is authoritative, who is allowed to use it, and what should happen when the evidence is incomplete or conflicting.

A curated pilot can hide those questions. Production cannot.

Infrastructure: the pilot hide the operating cost

A pilot can get surprisingly far on infrastructure that would never survive real usage.

Traffic is low. The data set is small. A developer can restart a failed job manually. Model latency is tolerated because only a few people are testing it. Retrieval can be re-indexed overnight. Nobody is asking what happens when the provider rate-limits a request halfway through a workflow.

Production removes those conveniences.

The system now has to deal with concurrency, retries, model and tool failures, cost limits, data freshness, background jobs, version changes, observability, and recovery without depending on the person who built the POC.

For retrieval-heavy systems, that may introduce ingestion pipelines, chunking and indexing jobs, freshness controls, hybrid retrieval, reranking, evaluation, and monitoring depending on the corpus and workload.

What looked like one step in the demo can become an operated subsystem with its own latency, quality, and cost profile.

Infrastructure answers whether the system can keep running. It does not answer whether it should be allowed to do everything it technically can.

Governance: permissions have to exist before the model sees the data

Governance often enters the conversation late because the pilot was built for a small group of trusted users.

Production changes the trust boundary.

The same assistant may now serve different teams, regions, customers, or roles. One user can open a document that another user should never retrieve. One workflow may be allowed to draft an action while another requires approval before anything changes in the source system.

Those controls cannot live only in a system prompt.

Authorization should be enforced before sensitive context reaches the model, and tool permissions should be scoped before the model decides which action it wants to take.

A permission filter applied after generation is not a permission filter. It is a leak with a delay.

Production governance therefore becomes part of the architecture: identity, retrieval filters, tool scopes, approval boundaries, audit records, data handling, and clear escalation when the system reaches a limit.

Even when the architecture is sound, there is another production dependency the technical diagram does not show: whether people actually change the way they work.

People and delivery: adoption fails before the dashboard shows it

A technically correct AI system can still fail in production because the workflow around it never changes.

During a pilot, users are usually motivated to test. They know the project team. They tolerate rough edges. When something fails, they report it directly.

Production users behave differently.

If the system creates more checking than it removes, misses important context, interrupts an established workflow, or requires users to maintain information in two places, people quietly work around it.

They return to spreadsheets. They copy the answer into another system. They ask a colleague instead. They stop trusting recommendations that need to be verified every time.

By the time declining usage appears in the dashboard, the trust decision may already have been made inside the workflow.

That is why adoption should not be measured only through logins or message counts. Look at whether the system actually removes steps, shortens cycle time, reduces rework, and becomes part of the process users already own.

That creates a measurement problem. Usage can tell you whether people touched the pilot. It cannot tell you whether the pilot deserves to become a production system.

Goals: the pilot needs a graduation metric

A pilot can be active without being valuable.

Messages sent, documents processed, users onboarded, and sessions completed are activity metrics. They tell you whether people used the system.

Production needs a different question:

What outcome has to improve enough for this system to deserve the cost and operating responsibility of production?

That may be time saved per case, processing errors avoided, faster case resolution, reduced manual entry, lower escalation volume, higher throughput, or cost per completed workflow.

The metric depends on the problem. What matters is that it exists before success is declared.

This is frequently left vague at kickoff because proving model feasibility feels more urgent than defining the threshold for production. The team proves the AI can do something, then discovers later that nobody agreed on what “good enough to scale” meant.

A pilot without a graduation metric can keep producing impressive demos indefinitely.

The cleanest way to prevent that ambiguity is to define the exit before the pilot begins.

Define the exit before the pilot begins

Before the first prompt is tuned, write down what would qualify the pilot for production.

Not just model quality.

The exit criteria should cover the things production will actually have to carry: business outcome, quality threshold, access control, latency, operating cost, failure behaviour, human escalation, monitoring, ownership, and the conditions under which the pilot should stop rather than scale.

For example, a pilot may need to demonstrate that:

• the defined business metric improves
• critical evaluation cases stay above an agreed threshold
• user-level permissions work correctly
• low-confidence cases escalate safely
• the expected production volume fits the cost model
• someone owns the system after launch

The values themselves will differ by use case. The important part is agreeing on them while the architecture can still change cheaply.

Then show your existing exit-criteria block.

pilot_exit_criteria:
  data_reality:    real sources, real mess, real versions, from day one
  infrastructure:  retrieval that scales, latency budget, cost per workflow
  governance:      permissions enforced before retrieval, audit trail complete
  operations:      evaluation set, traces, one named owner of correctness
  people:          the workflow that absorbs it, and who signs off
  goals:           the business number, its baseline, and what disproves it

Once those conditions are written down, the POC stops being an open-ended experiment. Every unresolved item becomes visible production work rather than a surprise after sign-off.

Eighteen questions that turn a pilot into a production plan

The purpose of a production-readiness review is not to give the pilot a score.

It is to expose the work that the demo did not need.

The questions below force the team to look beyond model output and examine the system that will have to operate around it: data, access, evaluation, infrastructure, recovery, ownership, cost, workflow adoption, and business outcome.

An unanswered question is not automatically a failure. It is a piece of production work that now has a name.

Then keep your existing 18-question diagram/checklist.

That changes the conversation. Instead of asking whether the pilot is “ready,” the team can identify exactly what remains, who owns it, and whether the expected value justifies completing it.

That is the whole difference between a stalled pilot and a roadmap. Same facts. One of them can be funded.

What changes when production is scoped from the start

The biggest difference between a pilot that stalls and one that moves forward is rarely one technical decision.

It is when production questions enter the project.

If data ownership, evaluation, permissions, cost, integration, recovery, and business success are discussed only after the demo works, the team discovers an entirely new project at the point it expected to deploy.

When those questions are part of the pilot scope, the same POC becomes evidence for a production decision.

This does not mean every pilot should be engineered like a full production system from day one. That would defeat the purpose of a POC.

It means the pilot should be built with visibility into what production will require, so the team knows which shortcuts are temporary and which decisions are intended to survive.

Key takeaways

1. A POC proves feasibility. Production proves the organisation can operate the system under real data, permissions, cost, and workflow conditions.
2. AI quality can degrade while infrastructure metrics remain healthy, which is why evaluation, sampling, and traces need to sit beside uptime and latency.
3. Data, infrastructure, governance, operations, workflow adoption, and goals all shape whether a pilot can graduate.
4. Activity metrics show that a pilot is being used. Outcome metrics show whether it deserves continued investment.
5. Production criteria should be written before the demo is judged successful, while the team can still design for them.
6. Any unanswered production-readiness question should become a scoped piece of work with an owner, estimate, and decision.

What this article cannot tell you

A framework can show you where production risk tends to hide. It cannot tell you which of those risks is actually holding back your system.

Two pilots can look almost identical in a demo and stall for completely different reasons.

One may have a retrieval problem disguised as a model-quality problem. Another may be technically ready but blocked by access controls. A third may have good architecture and no production case because the workflow economics do not hold at scale.

Those differences matter because the order of fixes matters.

Improving the model when the real issue is data readiness wastes time. Building more infrastructure before the production metric is agreed adds cost without answering whether the system should scale. Adding governance after the workflow is already designed can force changes much deeper in the architecture.

This article can help you recognize the signals.

It cannot tell you:

• Which constraint is actually limiting your pilot today?
• Which issues are symptoms of the same underlying problem?
• What needs to be fixed before production and what can wait?
• How much engineering work sits between the current POC and a production system?
• Whether the right decision is to scale, narrow the scope, redesign the system, or stop?

That requires looking at the system as it actually exists: its data, architecture, controls, evaluation, workflow, operating model, and the business outcome it is expected to produce.