An avalanche bulletin describes one area, for one day, at certain elevations and aspects. Anyone who goes off-piste reads it as information with an expiry date: the slope holding yesterday says little about today. I ski, and the comparison helps me explain a mistake I keep seeing in AI systems in production. It has limits: an eval has neither the coverage, the authority nor the stakes of a bulletin. The stories below, on the other hand, are real and come from my own projects.
The thesis: a check establishes something about one specific state. In software, a passing test, a successful verification or a green health check easily turn into permanent truths. Production work is deciding when that evidence expires, how you notice the state has changed and what happens next.
A passed check expires when the conditions that justified it change.
For CVing, an HR tech company in Milan, I built a WhatsApp agent that lets people apply for a job in conversation. For sensitive actions the candidate confirms their email with a code. In production a candidate got stuck: the code had expired mid-conversation and the system that manages it rejected it, while the agent still remembered “verified” and never asked for a new one. It kept telling them to check their inbox, and no email ever went out.
Every piece reported correctly what it knew. Put together, they trapped a person. I decided that a remembered verification has a lifespan, that when the system with the authority says no the local memory is wiped on the spot, and that a verification gets used in the same step as the action it protects, so it doesn't sit around getting old.
One rule lived in the model's instructions, recognising a code by its shape. It became a check in code. Whatever must always hold belongs where the model can't interpret it.
Before you retry, ask what may already have happened.
In the same review I went through every write asking what the upstream system guarantees when it fails. If requesting a new code fails, stop and retry. If saving an already successful verification fails, the verification stays valid and the problem goes into the audit log, described honestly. If the database doesn't answer, retry. If it answers that there's nothing there, that's an answer and you don't retry.
“Retry on error” squashes three different situations into one: nothing happened, everything happened, you don't know. Give every operation one of those labels and keep the uncertainty when there is some, in the state and in the audit.
A conversation that is orderly for the person can reach the server out of order.
The candidate lives a single conversation. The server receives messages one by one and can work on two at once, so two messages sent close together from the same number were fighting over the same data. I made messages from the same person queue up, while other candidates wait for nobody.
There was a second, less obvious part. When several quick messages are gathered into one turn, the model risked seeing them merged into a single text and losing which message said what. Now each one reaches the model with its position and its time. Find the thing whose state has to move forward in order, queue there, and when you combine several inputs keep the boundaries between them.
A new rule has to cover old data too.
In the same agent, the file names of uploaded CVs end up in front of the model, so they have to be treated as untrusted text. I added a filter at the entrance and clearly separated that name from the rest of the instructions. Then I applied the same filter to names already saved, every time they are read back.
That last step is the one people forget. Data saved before the rule is still there, and fixing the entrance doesn't touch it. For every new rule, look at where you read old data back and decide whether to migrate it, recheck it, isolate it or reject it.
A successful first step doesn't prove the second one happened.
In Yempik's company brain, approving a record worked. The next step, the one that makes it findable, failed silently, and search came back empty without an error. The success of the first step was acting as false evidence for the second. We added a check that keeps comparing approved records with findable ones, a search that says when it's behind, and a test that walks the whole path, from writing to searching.
One of the rules I work by covers this: every shortcut has a repair path. Write down durably what has to happen, and have something keep checking that it did.
Test from the environment that will do the work.
One of Yempik's sales routines relied on files on my laptop. In a local session those files are there, scheduled cloud runs can't see them. Every test run from the wrong session raised confidence without touching the weak spot. The classic “works on my machine”. Now every automation states what it reads, what it writes and where, and the test runs with the permissions and environment of whoever will execute it.
The same goes for health checks. A homepage that responds tells you the homepage responds. If the user's work goes through a database or an outside provider, the check has to travel that path, and a service that can't do its job should stop claiming it's available.
The most comfortable objection: the next model will make these problems go away.
Models really are improving. Look at the list, though: a verification remembered too long, an empty search, two messages racing, old data never rechecked, a cloud run without the right files. Spoiler: none of these depends on how smart the model is. A better model would have told the candidate to check their inbox more politely. The questions that matter sit outside the model: when does a piece of evidence expire, who notices unfinished work, who runs the job and with which permissions.
From Monday
- Give every important check a scope and a lifespan. When the authority says no, forget the yes you had in memory.
- Use a verification in the same step as the action it protects.
- Move the rules that must always hold into code.
- Before you retry an operation, decide whether it may already have happened.
- Queue work per person or thing whose state has to move forward in order, and keep the boundaries when you combine messages.
- For every new rule, also check where you read old data back.
- Write down durably what has to happen and keep checking that it did.
- Run your tests with the identity, permissions and environment that will really do the work.