Why AI pilots stall at the data layer

Somewhere between sixty and eighty per cent of enterprise AI pilots never reach production. The number moves depending on who is counting, but the direction never does. What is striking is how rarely the model is the reason.

We have been called into enough stalled pilots to notice that they fail in the same four places, and that all four are visible before the work starts. None of them are about model choice. All of them are about the data layer and the operating reality around it.

Here are the four questions. If you can answer all four, your pilot has a real chance. If you cannot answer one of them, that is not a reason to stop — it is a reason to sequence the work differently.

1. Can the data be joined without a human in the middle?

Almost every valuable use case needs two or more systems to agree about the same entity. A customer. A well. An asset. An invoice line. The pilot demo works because someone exported three files and matched them by hand over a fortnight. Production needs that match to happen every night, unattended, and to be right.

So the question is not "do we have the data." It is: is there a reliable key? And if not, is there a deterministic rule that produces one — or does joining require judgement? Judgement is not automatable at acceptable cost, and discovering that in month five is expensive.

2. Does the data exist at the resolution the use case needs?

This is where optimism does the most damage. A forecasting use case that needs hourly history does not survive on daily aggregates. A failure-prediction model needs enough labelled failures to learn from — and in a well-run operation, failures are rare, which is exactly the problem.

The specific traps we see repeatedly:

• Aggregation already happened. The historian compressed to fifteen-minute averages three years ago and the raw signal is gone. Your model cannot see what the storage policy deleted.
• History is shorter than the cycle you are modelling. Two years of data will not teach a model about a seasonal pattern with a three-year period, or about a market condition that last occurred in 2020.
• The labels are not labels. A maintenance work order says a pump was worked on. It does not say the pump failed, when it started failing, or why. Turning work orders into training labels is its own project, and it usually needs a domain expert for several weeks.

None of these are fatal. All of them change the sequence, the timeline and the cost. They should be discovered in a two-week assessment, not in month four of a build.

3. Who is allowed to see the answer?

This is the question that kills more knowledge-assistant projects than any technical constraint, and it almost never appears in the original scope.

An assistant grounded in your documents will happily surface a salary band from an HR file, a term from a contract the asking user is not party to, or a well result under a confidentiality agreement. The fix is not a filter on the output. It is inheriting permissions at retrieval time — the assistant retrieves only from documents the asking user could already open.

Which means the retrieval layer has to know, per document and per user, what the access rules are. If your document estate does not currently carry reliable permissions — and a shared drive that everyone has read access to does not — then you have a document governance project sitting in front of your assistant project. That is worth knowing in week two.

The hard part of a knowledge assistant has never been the model. It is retrieval quality and access control, and both are unglamorous engineering.

4. Who operates it on the Tuesday after go-live?

An AI system is not a deliverable, it is a running service with a cost curve and a quality curve, both of which drift. Somebody has to watch them.

Concretely, production means: an evaluation suite that runs on every change and catches regression before users do; version control over prompts, retrieval configuration and model selection, so a change is reviewable and reversible; spend monitoring with alerting, because inference cost scales with adoption and adoption is the thing you were hoping for; and a documented rollback path.

If none of that exists in the plan, the pilot was never scoped to reach production — regardless of how good the demo was.

What to do when the answer is no

The instinct is to treat a "no" as a blocker. It usually is not. It is a sequencing signal.

If the join is unreliable, the first project is not AI — it is an entity resolution and master data effort, which typically pays for itself in reporting accuracy alone before any model is trained. If the resolution is missing, change the storage policy today so that in twelve months you have what you need, and pick a use case that works on what you have now. If permissions are unreliable, start with a corpus where they are simple — public policies, published manuals, technical documentation — and prove the pattern before you point it at contracts.

What you should not do is proceed as though the answer was yes and hope the problem surfaces late enough to be someone else's. It always surfaces. The only variable is how much has been spent by then.

The honest version of the assessment

When we run an AI readiness assessment, most of the two weeks goes into exactly these four questions across your candidate use cases. The output is a scored list — value on one axis, feasibility on the other — and a recommendation for what to build first.

Sometimes that recommendation is not an AI project at all. Roughly a third of the time, the highest-value work turns out to be the data platform underneath, and the AI use cases become straightforward once it exists. We would rather say that in week two than bill for a year of finding it out.