AI proofs of concept stall for six recurring reasons, and almost none of them are modelling problems: there is no evaluation suite so nobody can prove the system is good enough to release; there is no defined behaviour for when the model is wrong; the running cost was never modelled; the demo ignored per-user permissions; latency is unacceptable outside a demo; and no team owns the system after launch. Each is a scoped engineering problem, and together they are usually three to six weeks of work rather than a restart.
Everybody quotes a statistic here — 70%, 80%, 90% of AI pilots never reach production, depending on which consultancy's survey you read. The number isn't the interesting part. The interesting part is that the pilots don't fail. They work, they impress people, and then they stop moving, and nobody can say precisely why.
Having been called in to restart several of these, the causes are boringly consistent. None of them are about the model.
1. Nobody can prove it's good enough
The demo was judged by watching it. Somebody typed five questions, the answers were good, everyone nodded. Then a director asks how often it's wrong, and the room goes quiet — because "we tried it and it seemed fine" is not an answer that authorises a release to customers.
Without a number, nobody can approve it. And without an eval suite there's no number, so the project enters a loop where more demos are requested and none of them resolve anything.
The unblocking move
Fifty to two hundred graded test cases drawn from real data, scored automatically, run on every change. That's usually three to five days of work and it converts an unanswerable question into a percentage somebody can sign off — or decline to, which is also progress.
2. There's no defined behaviour for being wrong
In a demo, a bad answer is a laugh and a retry. In production it's a support ticket, a wrong invoice, or a customer told something untrue by your software.
The question that stalls the release isn't "is the model good" — it's "what happens on the 4% of cases where it isn't", and a pilot almost never has an answer. Every path has to be decided deliberately:
- Output doesn't match the expected schema — retry, fall back, or fail visibly?
- The model is confident and wrong — is there a validation step, or a human in the loop for high-stakes actions?
- The provider is down or rate-limiting — does the feature degrade to something useful, or does the page hang?
- The answer isn't in the source material — does it refuse cleanly, or invent something plausible?
That last one matters more than teams expect. A system that says "I don't have that information" is far more valuable than one that always produces a paragraph, and it has to be built and tested for, not hoped for.
3. Nobody modelled what it costs to run
A pilot serves ten internal users. Production serves everyone. Somebody eventually multiplies the numbers and the project pauses "pending a business case", which is usually where it dies.
| Pilot | Production | |
|---|---|---|
| Users | 10 internal | 5,000 |
| Requests per day | ~50 | ~15,000 |
| Monthly model spend | Under $50 | $3,000 – $9,000 |
The fix is arithmetic before commitment, and then engineering: prompt caching cuts input token costs substantially on repeated context, routing simple requests to a small model instead of a frontier one changes the unit cost by an order of magnitude, and trimming retrieved context is usually the largest single lever. What an AI feature actually costs walks through the working.
4. The demo ignored who's allowed to see what
This is the one that kills internal knowledge assistants specifically, and it kills them late — usually in a security review, after months of enthusiasm.
The pilot indexed a folder somebody had access to. Production has to answer for every employee, using only what that employee may see, and enforce it at retrieval time rather than by filtering afterwards. That's a genuine re-architecture if it wasn't designed in: permissions have to be mirrored onto every chunk, queries filtered at the index, and the index re-synced when access changes. It's covered in more depth in the retrieval section of our AI integration page.
5. It's too slow to actually use
Eight seconds is fine when you're demonstrating and narrating. It's unusable when someone is trying to finish a task, and nobody notices during the pilot because the pilot audience is being patient on purpose.
- Stream the response. First token in under a second changes the perceived experience completely, even if the full answer still takes six.
- Cut the retrieval round trips. Retrieve once over a fused index rather than chaining three lookups.
- Escalate rather than default. A fast small model handles most cases and hands off only when it's unsure.
- Do the work before the click where you can. Pre-compute at write time instead of at read time.
6. Nobody owns it after launch
This is the real one, and it's why so many pilots stall even when the first five are solved.
An AI feature is not a static artefact. The provider will deprecate the model version. Pricing will change. Your data will drift under prompts tuned for how it used to look. The eval suite that passed in March will be failing by September while every test in CI stays green.
Whoever is asked to sign the release knows this, even if they can't articulate it, and they're being asked to take ownership of something with no named owner. Very sensibly, they don't. The pilot stays a pilot — which is safe, deniable and permanent.
How long it takes to fix
| Gap | Typical effort |
|---|---|
| Eval suite and a released accuracy number | 3–5 days |
| Defined failure paths and output validation | 2–4 days |
| Cost model and caching | 2–3 days |
| Permission-aware retrieval | 1–2 weeks |
| Latency work and streaming | 3–5 days |
| Monitoring and a named owner | 2–3 days, then ongoing |
Which is to say: a stalled proof of concept is usually three to six weeks from being shippable, not a restart. The reason it feels like a restart is that all six of these are invisible from the outside, so the only thing anyone can see is a demo that works and a project that doesn't move.
The last row is the one without an end date, and it's why our retainers exist as a named service with published prices rather than something improvised after the invoice clears.