Every stalled AI pilot we have been called into died the same way. Not loudly, in a post-mortem with names attached, but quietly, in the third quarter, when the sponsor changed roles or the budget cycle asked what the line item had returned. The demo was six months old by then. Nobody could say what the system did on a normal Tuesday, because it had never been given one.
The uncomfortable part is that the death was visible in week two. There are three tells, and they show up early enough to act on. This is the field guide we use in diagnostics.
Tell one: there is no evaluation suite
Ask a pilot team how they know the system works and listen for the shape of the answer. If the answer is a story about a demo that went well, the pilot is already in trouble. A production system is graded, continuously, against cases drawn from the operation it serves. If nobody has written that exam, nobody has defined what passing means, and the pilot will drift toward whatever impresses the room.
The exam is written before the student. If we cannot grade it, we do not ship it.
The evaluation suite is not a technical artifact. It is the specification, the contract and the early-warning system in one document. Its absence in week two predicts the third-quarter death with depressing reliability.
Tell two: no owner with a number
Every pilot has a sponsor. Almost no pilot has an owner: one person whose name is attached to a target metric that their leadership already tracks. Sponsors fund things; owners defend them. When the budget review arrives and the sponsor has moved on, a pilot without an owner has nobody to argue for it, and no number to argue with.
The fix costs nothing. Before the first line of code, write the target in a sentence: cost per claim down some percent, cycle time from thirty-four days to eleven, deflection above forty percent at equal satisfaction. If nobody will sign that sentence, the organisation has told you something important about the pilot, and it is better to hear it in week two than in month nine.
Tell three: a demo audience instead of a production queue
The third tell is the audience. A pilot that performs for executives is optimising for applause. A pilot that sits in the actual queue, handling real volume with a human reviewing its low-confidence cases, is optimising for survival. The difference is not subtle: one produces slideware, the other produces evidence.
If your pilot has been running for a quarter and has never touched the production queue, it is not a pilot. It is theatre. The kindest thing you can do is end the show and start the work: write the evaluation suite, name the owner, state the number, and put the system where the work actually happens.