Why most AI pilots never reach production, and why some ship errors when they do

Two Deloitte stories made a lot of Australian businesses think about AI more carefully in the last year.

The first was a refund. In October 2025, Deloitte Australia agreed to repay the final instalment of an A$440,000 contract with the Department of Employment and Workplace Relations. A report it had delivered contained references to academic work that doesn't exist and a quote attributed to a Federal Court judgment that the judge never wrote. The corrected version disclosed that a generative AI tool had been used in drafting. The department said the substance and recommendations of the review were unchanged.

The second was a survey. Deloitte's 2026 State of AI in the Enterprise research found that only 28% of Australian respondents had moved 40% or more of their AI pilots into production.

These look like opposite problems. One is AI output shipping without enough checking. The other is AI work that never ships at all. In our experience they have the same cause: nobody defined, measured and owned what "good enough" means for the specific job the AI is doing.

Why pilots stall

Most AI pilots we're asked to look at stalled for one of these reasons.

• The goal was "use AI", not a workflow. Without one specific job and one person who owns it, there's no way to say whether the pilot worked, so it never gets approved for production.
• It was judged from a demo. It answered ten hand-picked questions well. Nobody knows how it does on the thousand real ones, so nobody is willing to put their name to switching it on.
• Nobody worked out what it costs to run. Model and infrastructure costs at real volume weren't estimated, so finance can't approve it.
• Risk had no owner. Legal, security or compliance asked reasonable questions about data and errors, and there was no one to answer them.
• It was never connected to anything. The pilot lived in a chat window. The real work happens in your CRM, finance system or ticket queue, and integration was out of scope.

None of these is a model problem. Better models don't fix them.

Why AI errors ship

The refund story is a useful reminder that the risk isn't only AI being wrong. It's AI being wrong in a way that looks right. A fabricated reference is formatted perfectly. A made-up quote reads like a real one. The output passes a quick read, and a quick read is often all it gets.

People get worse at checking work the more often it's right. When an assistant is right 95% of the time, reviewers start skimming, and the 5% gets through. "A person checks it" isn't a control unless you've designed what they check, how, and how long it takes.

What a production-ready AI feature has

Before we call an AI capability ready to ship, we want each of these in place.

One workflow and one owner

A specific job, such as extracting fields from supplier invoices or drafting replies to a category of support ticket, with a named person who knows what a correct result looks like and signs off on it.

Written acceptance criteria

What counts as correct, what counts as harmful, and what the system should do when it isn't sure. For anything involving facts, sources or numbers, "say you don't know" is a required behaviour, not a failure.

An eval set built from real examples

A scored set of real inputs with known right answers, including the awkward cases and the ones where the right answer is "no". It's run on every change to the prompt, model or data, so quality can't drop without anyone noticing. We've written in detail about how we build evals before launch.

Checks that don't depend on someone reading carefully

Where something can be verified by code, it should be. Does each cited source exist and contain the quoted text? Does the extracted amount match the document exactly? Is a required field missing? These checks don't get tired or skim. They would have caught invented references before anyone had to.

Human review designed for the cost of a mistake

People stay in the loop where an error is expensive, and the review is made easy: the source is shown next to the output, low-confidence results are flagged, and approving or correcting takes seconds. Corrections feed back into the eval set.

A running cost and an owner after launch

A per-request and per-month cost estimate from real usage, a monitoring plan, and someone responsible for reviewing quality on a schedule, not only when something breaks.

How we take a pilot to production

This is the reason our AI work starts small and measured.

AI Opportunity Sprint (2–3 weeks, A$5k–15k). We pick one workflow with its owner, build a working prototype on your real data, and write the first eval set. You leave with a scorecard showing how accurate it is, where it fails, what it would cost to run, and where people need to stay in the loop. If the honest answer is "don't build this", that's the recommendation. More detail in what the Sprint includes.

Production Pilot (A$15k–150k, 3–12 weeks). We integrate the feature into the systems where the work actually happens, put the eval set in as a regression gate, build the review workflow, and set up monitoring for quality, cost and drift.

Throughout, AI speeds up the engineering, and senior engineers review and approve every change before it ships. That applies to the AI feature itself as much as to the code around it.

FAQ

Is it safe to use generative AI for reports and documents?

It can be, for drafting, summarising and searching, as long as anything presented as fact is checked against its source, ideally automatically. The risk is in publishing output that hasn't been verified, not in using the tool.

How do we know if our AI pilot is ready for production?

You should be able to show a scorecard on real examples that meets criteria the workflow owner agreed, a production cost estimate, and a defined review process for the cases where mistakes are costly. If any of those is missing, that's the next piece of work.

Should AI make decisions without human approval?

For low-cost, reversible actions with good evals behind them, it can. For anything financial, legal, safety-related or customer-facing that's hard to undo, keep a person in the loop and make their review quick.

Why do AI pilots fail?

Usually because there was no clear workflow, owner or measure of success, not because the model wasn't good enough.

---

Have an AI pilot that's stuck, or one you're not sure is safe to switch on? Book a scoping call. We'll look at what you have and tell you what it would take to measure it and get it into production, or whether it's worth it at all.