The Demo Worked. Six Months Later, It Is Still a Pilot

Every enterprise I walk into has a proof of concept that works. A chatbot over internal documents. A copilot that summarises reports. An agent that clears a repetitive queue. The demo lands, the room nods, the budget gets approved.

Then the calendar moves and the project does not. Requirements shift. Security appears with a list. Costs get vague. The system that was crisp on curated files turns unreliable on the real thing. Nobody kills it. It just keeps getting called a pilot until people stop asking.

I have watched this from inside the engineering room more times than I would like. And the part that still surprises people: the pilots that stall are rarely the weak ones. They stall because the demo succeeded early enough to convince everyone the hard part was over.

There is also a pattern I did not expect. In almost every stalled pilot I have reviewed, the thing that actually killed it was the last thing anyone thought to check. I will come back to it, because you will not like where it sits.

Short answer: A POC and a production AI system are different classes of system. The POC answers one question, can the model produce useful output, inside an environment engineered to be forgiving. Production answers a harder one, can this organisation operate the system safely, reliably and economically at scale. Six lenses decide the second answer: data reality, infrastructure, governance, operations, people and delivery, and goals. A stalled pilot is normally failing three or four at once, and almost none of that work was scoped when the demo was approved.

The number nobody quotes

You have seen the headline. MIT's Project NANDA report, The GenAI Divide: State of AI in Business 2025, found roughly 95% of GenAI pilots produced no measurable impact on the profit and loss statement. It got quoted into meaninglessness within a week.

The figure underneath it is the one that should worry you, and it is almost never cited. For enterprise-grade tools, about 60% of organisations evaluated them. Around 20% reached a pilot. About 5% reached production.

Read that as a funnel and it says something specific. The idea is not where projects die. Nor is the evaluation. Two thirds of the mortality happens after someone has already decided the thing is worth doing. The crossing is the graveyard.

S&P Global Market Intelligence found the pattern accelerating rather than improving: the share of companies abandoning most of their AI initiatives climbed from 17% to 42% year over year, with the average organisation scrapping 46% of proofs of concept before production. Cost, data privacy and security topped the obstacle list.

And Gartner predicted over 40% of agentic AI projects would be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls.

Now a small thing that tells you how carelessly this whole field reads its own evidence. Gartner published that prediction in June 2025. Most of the coverage circulating today drops the date and presents it as a fresh 2026 finding. If a statistic in an AI article is undated and unlinked, it is decoration. The same discipline you are about to apply to your pilot applies to the research about pilots, and almost nobody applies either.

The demo was never the system

A POC is not a small production system. It is a different artifact with a different job, and it is completely honest about that job right up until someone mistakes it for a foundation.

It runs on a folder someone cleaned by hand. It serves the people who made it. It answers the questions they thought to ask. Everyone sees everything, because permissions were out of scope. Someone watches it during the demo, which counts as monitoring. Cost never comes up, because the volume is a rounding error.

Every one of those is a deliberate simplification, and a correct one. Each is also a system that does not exist yet.

The POC proves the idea is possible. Production proves your organisation can run it. Those are not the same claim, and only one of them was tested.

Your dashboard is green. Your system is broken.

Here is the property I wish someone had explained to me before my first AI system went live. It is the reason the standard operational playbook misfires, and it sits underneath everything else in this article.

Conventional software fails loudly. AI systems fail quietly.

When a normal service breaks, it tells you. A 500 comes back. A queue backs up. Latency spikes, an alert fires, someone gets paged, the incident starts. Every operational instinct we have inherited, dashboards, uptime checks, error budgets, is calibrated to catch a system that announces its own failure.

An AI system fails by returning a fluent, confident, well-formatted, entirely wrong answer. With a 200 status code. In 800 milliseconds. Your dashboard goes green. Your error rate stays flat. Your latency looks excellent. The system is broken and every instrument you own is reporting perfect health.

Sit with that for a second, because the consequences are not small. It means nobody gets paged. It means quality can decay for weeks before a human finally escalates, by which point trust is gone and the pilot is already politically dead. It means the failure is invisible precisely to the people funding it.

You cannot alert on an answer that was merely wrong. You have to go looking for it, deliberately, on a schedule, with evaluation sets and sampling and traces. That is not a tool you install. It is a practice you staff.

Which is why, when a client asks us to review a stalled pilot, the first artifact we ask for is not the architecture diagram. It is the evaluation set. If it does not exist, we already know most of what we are going to find, and so does everyone in the room the moment the question is asked.

Six lenses. Your pilot is failing three of them.

We do not start with the model, the vendor or the prompt. We look through six lenses. A stalled pilot is always failing at least one, and usually three or four at once, which is exactly why the stall feels sudden when it was actually gradual.

Data reality: the pipeline was a person

The POC worked because an engineer hand-prepared a small, clean dataset. That person was the pipeline. Nobody wrote that down, because at the time it was not a decision, it was a Tuesday. At real scale you meet duplicates, superseded documents with no version marker, scans that stay images until something runs OCR over them, missing metadata, and two regions using different words for the same field. MIT NANDA named the core barrier a learning gap: tools that do not retain feedback, adapt to context or match real workflows. Data reality is where that gap physically lives.

Infrastructure: you scoped a step and received a subsystem

A demo index over a few hundred documents is not a data pipeline. Production retrieval needs ingestion, chunking, dual indexing, freshness rules, reranking, evaluation and observability, and it becomes an independent system with its own failure modes and its own on-call. Cost lives here too. POC economics are one model, short prompts, few requests. Production adds traffic, longer contexts, retrieval, retries, multiple models and agent workflows, and those compound rather than add. Teams meet the real curve on an invoice instead of a dashboard. We covered how to engineer this layer in the RAG architecture guide.

Governance: a leak with a delay

The POC asked whether the model could answer. Production asks whether this user is allowed to receive this answer. Different systems entirely. Authentication, authorisation, audit logging, residency, sensitive data handling: none of it is hard in isolation, all of it is brutal to retrofit into a retrieval layer that assumed one flat index. And the shortcut teams reach for does not work. A permission filter applied after generation is not a permission filter. It is a leak with a delay.

Operations: volunteers, not owners

This is where silent failure collects its debt. Who reviews hallucinations? Who versions prompts and retrieval configs? Who notices a regression after a model update? Who watches cost drift? During a pilot, the honest answer is whoever cares most. That is not an owner. That is a volunteer, and volunteers get reassigned to the next quarter's priority. Production needs one named team accountable for whether the system is still correct, not merely whether it is still running.

People and delivery: nobody files a ticket saying they stopped trusting it

A system nobody uses is indistinguishable from a system that does not work. If the output lands somewhere the workflow does not touch, or asks a specialist to re-verify everything by hand, adoption dies quietly and politely. There is no error, no complaint, no signal. MIT NANDA found the shadow of this everywhere: people reach for personal tools while the sanctioned pilot sits unused. Your usage chart will show the decline about two months after the decision was actually made.

Goals: the lens that actually killed it

This is the one I promised at the start, and here is the uncomfortable part. It is the first lens chronologically, the easiest to answer, and the one almost every stalled pilot skipped, because at kickoff it felt like paperwork standing between the team and the interesting work.

Ask a stalled pilot how it is doing and you get prompts served, users onboarded, documents indexed. Those are activity metrics. They prove the thing is running. Leadership funds outcomes: time saved per case, errors avoided, cases closed without escalation, cost per completed workflow.

So the pilot walks into the budget review with evidence that it exists and no evidence that it matters. And here is why that is fatal rather than merely awkward. Nobody in that room says the project failed. There is nothing to point at. It simply does not get prioritised, this quarter, and then the next one. That is what a stall actually is: not a decision anyone made, but a decision nobody could make, because the number that would have made it obvious was never defined.

A pilot rarely dies of one cause. It dies of three lenses nobody looked through, and one nobody wrote down.

The one meeting that saves six months

The cheapest intervention I know costs a single meeting, held before the POC rather than after it. Decide what would make this pilot graduate, in writing, while everyone is still optimistic and nothing is political yet.

pilot_exit_criteria:
  data_reality:    real sources, real mess, real versions, from day one
  infrastructure:  retrieval that scales, latency budget, cost per workflow
  governance:      permissions enforced before retrieval, audit trail complete
  operations:      evaluation set, traces, one named owner of correctness
  people:          the workflow that absorbs it, and who signs off
  goals:           the business number, its baseline, and what disproves it

Nothing there is exotic. The entire value is in the timing. Written before the demo, it is a plan. Written after the demo, it is a post-mortem with better formatting.

Eighteen questions most teams have never been asked

The six lenses become eighteen questions. No maturity model, no scoring, no chart. Tick only the boxes you could defend out loud in front of the people funding this. Most teams get further down the list than they expected before the first honest blank appears.

The point is not to pass. Almost nobody passes on a first attempt, including teams whose pilot is genuinely good and eventually ships. The point is that a box you cannot tick stops being a vague sense that the project is dragging, and becomes a scoped piece of engineering work with an owner and an estimate.

That is the whole difference between a stalled pilot and a roadmap. Same facts. One of them can be funded.

What the 5% do that you are not

The research is unusually consistent here, and it is not flattering to the way most pilots are run.

MIT NANDA found the divide was determined by approach rather than model quality or regulation, and that externally partnered, deeply customised systems reached deployment roughly twice as often as internal efforts. That second finding reads like a vendor pitch. I read it as a staffing observation. These failure modes are specific, quiet, and mostly invisible to a team encountering them for the first time, which is not a criticism of the team. It is the nature of failures that do not announce themselves.

None of this makes your POC a waste. It did its job: it proved the idea deserves the engineering. The error is treating a feasibility study as a finish line.

Key takeaways

1. The mortality is after approval, not before it. MIT NANDA tracked roughly 60% evaluating, about 20% piloting, about 5% reaching production.
2. AI fails quietly, with a confident wrong answer and a 200 status code, so inherited monitoring reports health while quality decays and nobody gets paged.
3. Six lenses decide it: data reality, infrastructure, governance, operations, people and delivery, and goals. A stalled pilot is failing three or four at once.
4. Goals is the lens that kills the most pilots. It is the easiest to answer and the one most often skipped, because at kickoff it feels like paperwork.
5. A stall is not a decision anyone made. It is a decision nobody could make, because the number that would have settled it was never defined.
6. Any lens you cannot answer is not a blocker. It is the next piece of engineering work, scoped and owned.
7. Check the date on any statistic about AI failure. Undated numbers are decoration.

What this article cannot tell you

Everything above is the map. It is the real map, and it is yours. But read the eighteen questions again and notice what just happened: you can see the shape of the problem, and you still cannot name your own answer.

That is not a gap I engineered into the writing. It is structural. An article cannot see your corpus, your permission model, your traces or your invoice. So three things stay out of reach no matter how carefully you read.

Which lens is actually failing. Every stalled team I have met had a confident theory. Roughly half were wrong, and the theory usually pointed at the lens they understood best rather than the one that was broken.

What it costs to clear, and in what order. Three lenses failing at once is normal. Which one you fix first changes the timeline by months, because some unblock others and some are dead ends until the data reality is handled.

Whether it is worth clearing at all. Some pilots should be stopped. That verdict needs someone with no stake in the answer, and nobody inside the project is in a position to give it.