Rollback plan for an AI workflow: what can still be put back after it acts.

Somewhere in the paperwork for a new system there is a line about rollback, and the head of operations reads it as a promise that any mistake can be undone. Say the system places replenishment orders for a grocery chain overnight, and one night it misreads a supplier's pack size and orders cases where the stores wanted single units. Somebody puts the previous version back before breakfast. The orders are still with the supplier.

So a rollback question gets asked twice, once about the code and once about what the system has already done, and a plan that answers only the first leaves the grocery chain on the phone to its supplier.

Putting the code back

For code, the rollback worth having is the boring one. On octyn.co it is a revert and a push, the same path as any deploy, so there is no separate emergency procedure for somebody to get wrong at two in the morning, and a build that does not compile never replaces the one running. Mooney, the expense app OCTYN runs, ships through an app store, so a bad build halts partway through a staged rollout instead of being recalled from every phone.

Then ask whether anyone has done it. The first attempt always uncovers something dull, like a migration that does not reverse or credentials nobody has, so it is worth asking who last ran the rollback, and when.

Actions still waiting to happen

Most of what an operations system does sits in a queue for a while before it acts, and that wait is the easiest place to undo anything. In the outreach system OCTYN runs for an independent craft retailer, rollback is per action: a queued message can be changed or cancelled up to its send time, and the changed version is the one that goes. That mattered on 12 August. A rerouting step had queued a cold first message to a third person at a company already talking to the retailer, and it was cancelled sixteen minutes before it would have gone out.

For the grocery chain, the same idea is an order that waits with a time on it, where a buyer can change the quantity or pull it before the supplier sees anything. A dry run on the first night would have caught the pack size on paper.

Data the old version cannot unwrite

If the system wrote something wrong to the database, going back to the old code leaves the wrong rows where they were. Deletes are worse. One of OCTYN's own databases had two links set to cascade, so removing one record quietly removed linked records that had taken days to collect. Both links now refuse. A delete that would take other data with it is blocked, and it says what it is blocking.

Before the system goes live, hand this to whoever will run it and ask for an answer in every row.

If the answer in the bottom row is the backups, ask when somebody last restored one.