AI demo vs production: what week six brings that the boardroom never saw.

The parts manager chose the ten job notes for the demo, and chose them fairly. The dealer group here is made up, and so is its boardroom, where a system reads what a technician wrote on a job and finds the right part at the right branch while the biscuits go round. It gets all ten. Then the operations director asks when it can go live, a reasonable thing to ask of something that has just been right ten times.

The notes were real and the parts were correct. They were also what a careful person picks when asked for examples: notes somebody could read, from the branch that person runs.

What week six has that the boardroom did not

Six weeks in, the system is reading every note from every branch.

What got checked in the boardroom was whether the model could read a job note. By week six the question is whether the system knows how this group runs, such as which branch kept its old stock codes after being bought, and who to ask when a note makes no sense. Each row in that table is an ordinary day at a dealer group.

The failure that reports success

A wrong part on the bench gets noticed by lunchtime. The worse case is the system telling you all went well. OCTYN runs an outreach system for an independent craft retailer, and the model driven sender it tried first errored, sent the wrong thing, then reported a send that had not happened. Its replacement has no intelligence in it: a small service that checks what is due every minute and confirms each send.

That system's volume is small, and the calendar still caught it out. On 10 August 2026, 115 follow ups came due across a weekend and nothing picked them up, since the check for what was due only looked at today, and Monday already held 93 against a cap of 60. The scheduler now moves overflow forward when it creates a follow up, and moves weekend dates to Monday. A demo held on a Wednesday has no way of finding that.

What to ask for before it goes live

Ask to run it on a bad week of your own records, picked by somebody other than whoever picked the demo set. Watch what it does with the note it cannot read. You want it to hand that note to a person and say why, because whether to order the part on a guess is a call for the parts desk.

Then ask what somebody reads each morning, and what it says on the day the provider is down. OCTYN's write up of the retailer's system says the daily summary should have been built first, since without it the operator has nothing to read. Every guard there sits in the operating notes beside what it prevents, and those notes are what the next person has when the parts manager goes on leave. For the longer checklist, see the page on what production ready means.

OCTYN has no number for how many demos fall apart after launch, so this post does not offer one.

At the made up dealer group, Dave retired in the spring, and nobody has worked out what last time was.