Why AI pilots stall: a gate based model for taking AI from pilot to production

Background

An AI pilot is easy to start. A team picks a promising task, connects a hosted model, prepares a few dozen examples and shows a demo within weeks. The demo usually works, because demos are built on the examples that work.

The hard part comes after. The pilot has to run on real data, inside real systems, used by people who did not ask for it, with someone answering for it when it gets something wrong. That is where many pilots stop moving. They are not cancelled. They simply stay in a "phase two" that never gets a budget or a date.

For a business owner or CFO, a stalled pilot is a cost twice over. There is the money spent, and there is the lost confidence that makes the next sensible AI project harder to approve.

Approach

This paper is based on analysis of public documents and the practices we use in delivery. We read the governance and risk material that organisations increasingly cite when they adopt AI, chiefly the NIST AI Risk Management Framework 1.0, ISO/IEC 42001:2023 and, for organisations with European exposure, Regulation (EU) 2024/1689 (the EU AI Act). We compared what those documents expect of an AI system with what a typical pilot plan actually contains.

We also drew on the delivery checklists we use when we take an AI system from assessment to managed operations. No client data, names or results are used, and the paper makes no estimate of how many pilots fail across the market.

The findings are therefore qualitative. They are reasoned from the gap between what production requires and what pilots usually plan for.

Findings

1. The success measure is missing or vague

A pilot needs one sentence that says what good looks like. "Reduce manual data entry for supplier invoices posted to Tally" is a start. "Post at least 70 percent of supplier invoices without manual typing, with no increase in posting errors" is a measure. Without it, every review becomes a debate about impressions, and the loudest opinion wins.

2. Pilot data is not production data

Pilots tend to run on samples that someone has cleaned, often without meaning to. Real inputs include scanned PDFs at an angle, emails with three attachments, WhatsApp photos, mixed English and Gujarati, and records with missing fields. A system that has never seen these will behave differently when it meets them, and the team only finds out after launch.

3. Access is assumed, not secured

Reading from an ERP, a CRM or a shared drive in a pilot is often done with a personal login or an export. Production needs a service account, an approved integration, a data processing basis under the Digital Personal Data Protection Act, 2023 where personal data is involved, and sign off from whoever owns the system. Each of those can take weeks. If nobody starts them during the pilot, they become the reason the pilot stalls.

4. There is no evaluation set

Without a fixed set of real cases and expected results, nobody can say whether the system is improving or getting worse when a prompt, model or data source changes. Teams fall back on trying a few examples by hand, which feels like testing but is not.

5. Nobody owns operations

Models are updated and retired by their providers. Data drifts. New products and new suppliers appear. An AI system needs someone to watch its quality, cost and failures every week. When the pilot team moves on and no operations owner is named, the system degrades quietly and users stop trusting it.

6. Risk review arrives too late

Security, privacy and legal review are often booked for the week before go live. Questions that are cheap to answer early, such as where prompts and outputs are stored, or whether a human approves actions above a value, become expensive late. The NIST framework and ISO/IEC 42001:2023 both treat risk management as something that runs through the whole life of the system.

A framework

We call this the Five Gate Model. Each gate has pass criteria written before work starts. A sponsor can check every criterion by asking for a document or a demonstration, without needing to read code. Passing a gate releases the budget for the next stage. Failing a gate means you fix the gap or stop.

Gate 0: Problem

• One use case, described in the words of the people who do the work today.
• A success measure in a single sentence, with a number and a guardrail.
• A named business owner who will use the result and a named sponsor who will pay for it.
• A rough estimate of value and running cost, with the reasoning written down.

Gate 1: Data and access

• A sample of real inputs, including the awkward ones, reviewed by the team.
• Confirmed technical route to every system the AI needs to read or write, with the owner of each system agreeing in writing.
• A short note on personal data: what is processed, on what basis, and where it goes, including any model provider.

Gate 2: Evaluation

• An evaluation set of real cases with expected outcomes, agreed by the business owner.
• A target score on that set, tied to the success measure from Gate 0.
• The pilot meets the target on the evaluation set and on a period of live shadow use, where the system runs alongside people but does not act alone.

Gate 3: Operational readiness

• A named operations owner and an agreed support arrangement.
• Monitoring for quality, cost, latency and failures, with alert thresholds.
• Human approval in front of any action above an agreed risk level.
• Security and privacy review completed, including the relevant items in the OWASP Top 10 for Large Language Model Applications.
• A rollback plan: how to switch the system off and return to the manual process in under a day.

Gate 4: Value

• The success measure is met in production for an agreed period, usually several weeks.
• Users have been trained and their feedback recorded.
• A decision is taken to scale, hold or retire, and recorded with its reasons.

The order matters. Gate 1 comes before any serious build because data access is the most common reason a promising pilot cannot proceed. Gate 3 sits before launch, not after, because operations is where unowned systems decline.

Recommendations

1. Write the Gate 0 page before you approve any spend. If the sponsor cannot produce a one sentence success measure, the pilot is not ready, and that is a cheap thing to learn.
2. Start access and privacy work in week one. Treat integration approvals and data protection notes as part of the pilot itself.
3. Build the evaluation set from real, messy cases. Ask the people who do the work for the examples they find hardest. Twenty difficult cases are more useful than two hundred easy ones.
4. Name the operations owner at the start. If nobody will own the system after launch, either plan a managed service or do not build it.
5. Plan for a stop. Agree at Gate 0 what result at Gate 2 would end the project. A stop at Gate 2 that costs a few weeks is a good outcome compared with a system that limps along for a year.
6. Keep one use case per pilot. Bundling three use cases into one pilot makes every gate harder to pass and every result harder to read.

Limitations

This is a position paper. We have not measured pilot outcomes across organisations, and the failure points we describe are drawn from public frameworks and our own practice, which may not reflect every sector. The model assumes a sponsor with authority to stop work, and some organisations have no such role. The model is also tuned for task automation and decision support. Research projects, where the aim is to learn, need looser gates. Finally, the gates add some paperwork. For a very small pilot run inside one team with no personal data, a lighter version with Gates 0, 2 and 4 may be enough.

Sources

• NIST AI Risk Management Framework 1.0
• ISO/IEC 42001:2023
• Regulation (EU) 2024/1689 (EU AI Act)
• Digital Personal Data Protection Act, 2023 (India)
• OWASP Top 10 for Large Language Model Applications