# AI demo vs production: what week six brings that the boardroom never saw

The gap in AI demo vs production is mostly the cases left out: a demo runs on clean examples somebody picked, and production gets whatever the operation produces that week, including the days you would never choose to show. Before signing, ask to see it run on a bad week of your own records.

Date: 2026-11-07
Reading time: 3 min
Author: OCTYN
No. 25

The parts manager chose the ten job notes for the demo, and chose them fairly. The dealer group here is made up, and so is its boardroom, where a system reads what a technician wrote on a job and finds the right part at the right branch while the biscuits go round. It gets all ten. Then the operations director asks when it can go live, a reasonable thing to ask of something that has just been right ten times.

The notes were real and the parts were correct. They were also **what a careful person picks when asked for examples**: notes somebody could read, from the branch that person runs.

## What week six has that the boardroom did not

Six weeks in, the system is reading every note from every branch.

| what changes | in the boardroom | in week six |
|---|---|---|
| volume | ten notes, one at a time | every branch, and Monday's queue |
| the input | complete notes, picked by hand | same as last time, ask Dave |
| the people | the parts manager, in the room | the parts manager on leave |
| the provider | answering | down at ten on a Thursday |

What got checked in the boardroom was whether the model could read a job note. By week six **the question is whether the system knows how this group runs**, such as which branch kept its old stock codes after being bought, and who to ask when a note makes no sense. Each row in that table is an ordinary day at a dealer group.

## The failure that reports success

A wrong part on the bench gets noticed by lunchtime. The worse case is the system telling you all went well. OCTYN runs an outreach system for [an independent craft retailer](/casestudies/craft-retail-brand-sales), and the model driven sender it tried first errored, sent the wrong thing, then **reported a send that had not happened**. Its replacement has no intelligence in it: a small service that checks what is due every minute and confirms each send.

That system's volume is small, and the calendar still caught it out. On 10 August 2026, **115 follow ups came due across a weekend and nothing picked them up**, since the check for what was due only looked at today, and Monday already held 93 against a cap of 60. The scheduler now moves overflow forward when it creates a follow up, and moves weekend dates to Monday. A demo held on a Wednesday has no way of finding that.



## What to ask for before it goes live

Ask to **run it on a bad week of your own records**, picked by somebody other than whoever picked the demo set. Watch what it does with the note it cannot read. You want it to hand that note to [a person](/glossary/human-in-the-loop) and say why, because whether to order the part on a guess is a call for the parts desk.

Then ask what somebody reads each morning, and what it says on the day the provider is down. OCTYN's write up of the retailer's system says the daily summary should have been built first, since **without it the operator has nothing to read**. Every guard there sits in the operating notes beside what it prevents, and those notes are what the next person has when the parts manager goes on leave. For the longer checklist, see [the page on what production ready means](/compare/production-ready-ai-system).

OCTYN has no number for how many demos fall apart after launch, so this post does not offer one.

At the made up dealer group, Dave retired in the spring, and nobody has worked out what last time was.

## Related

**Case studies cited**

- [Brand sales for an independent craft retailer](https://octyn.co/casestudies/craft-retail-brand-sales)

**Terms used here**

- [Human in the loop](https://octyn.co/glossary/human-in-the-loop)

**The longer argument**

- [What production ready means for an AI system](https://octyn.co/compare/production-ready-ai-system)

**Read next**

- [How much does an AI system of your own cost, and what it costs to change later](https://octyn.co/blog/how-much-does-an-ai-system-of-your-own-cost-and-what-it-costs-to-change-later)
- [Who owns the code when an agency builds your system, and what to get in writing](https://octyn.co/blog/who-owns-the-code-when-an-agency-builds-your-system-and-what-to-get-in-writing)
- [AI agent vs workflow automation for small business: most of the work wants rules](https://octyn.co/blog/ai-agent-vs-workflow-automation-for-small-business-most-of-the-work-wants-rules)
- [How to scope an AI project so it ships: write down what it will never do](https://octyn.co/blog/how-to-scope-an-ai-project-so-it-ships-write-down-what-it-will-never-do)

More on build or buy: [Build or buy](https://octyn.co/blog/topic/custom-ai-systems)

Canonical: https://octyn.co/blog/ai-demo-vs-production-what-week-six-brings-that-the-boardroom-never-saw