AI inside a regulated process: the checklist.

The problem

In a regulated process, nobody is buying an answer. They are buying a defensible way of reaching it. The model is one step in a chain that already has owners, records and audits, and the chain does not bend to fit the model. The model has to fit the chain.

That changes what “done” means. A system is done when a compliance officer can explain, for any single decision, what the model saw, what it suggested, who decided and why. Everything below follows from that sentence.

The constraints

We were adding a step, not redesigning a process. The same roles kept signing, the review had to happen in the tool the team already had open, and nobody had time to learn a second one. Whatever the model produced had to be legible to a person who would never read the code.

What we tried

The first version showed a suggestion with no sources and a confidence score. Reviewers asked where the number came from; nobody could say.

What failed

Confidence without provenance. A score is not an explanation, and a compliance officer cannot sign a percentage.

What worked

What worked was answering these seven questions before the first model call, and writing the answers where the team could read them.

1. Where does the data live, and where does it go? Every document and every prompt has a path. Draw it. If the path leaves your environment, you need a reason and a contract, not a default.
2. What does the model never see? Some fields are out of bounds by law, some by policy. Remove them before the model, not after.
3. What is the model allowed to decide? Almost always: nothing. It reads, ranks, drafts and flags. The person decides. Write that boundary into the interface, not into a training deck.
4. How is every suggestion traceable? The answer shows its sources. The log keeps the version of the model, the prompt and the inputs. A year later you can reproduce the decision.
5. What happens when it is unsure? The system says so and routes the case to a person. Silence is not an option; guessing is worse.
6. Who reviews the flagged cases, and how often? Name the role. Put it in the rota. A review queue nobody owns is a queue nobody reads.
7. How do you switch it off? A feature flag, a fallback to the manual process, and a rehearsal. If turning it off takes a release, it is not under control.

What we learned

The people in the process get faster at the part they were already good at, and the record of every decision gets better than it was before the model arrived. Auditors like it. The team likes it. Nobody has to trust a black box, because there is no box: there is a documented step with a person at the end of it. Reviewers stopped asking where the number came from once every suggestion arrived with its sources.

What we would do differently

We would put traceability in before accuracy and rehearse the off switch in the first week, not the last.