DeepSWE (si apre in una nuova scheda), an agentic coding benchmark with tasks written from scratch and hand-written verifiers, put a number on something people working with agents have felt for a while. On SWE-Bench Pro, frontier models sit within a band of about 30 points. On DeepSWE, with tasks nobody has seen before, the same models spread across about 70. When I commented on it, my conclusion was this:
A score that doesn't look like your work tells you nothing about your work.
Evaluate on your own work, own the infrastructure, the core logic and the verification, and treat models and harnesses as replaceable parts. Working on it, I understood better what owning means. Owning a harness doesn't mean building one. It means owning the things that let you swap a model or a tool without losing control: state, authority, operating rules and verification.
The layer you own sits above the native tools
At first I wondered when owning the harness stops paying off, meaning when a native product already knows enough about your environment. For coding agents the answer is: now. Claude Code and Codex edit code, isolate execution and hook in checks better than I would. Rebuilding that would be wasted work.
So the layer I own moved up a level. It's my own versioned rules: which model does which work, where every result lands, what a worker is allowed to do, what I check myself before accepting. The tools underneath can be swapped. The same reasoning holds for products: identify the rules you have to control yourself and use what already exists underneath.
The verifier is part of the product
The DeepSWE authors checked the verifiers inherited from earlier benchmarks, and found plenty of tests that reject correct implementations or accept incomplete work. A pass looks objective, and it depends entirely on whoever wrote the check.
And a score holds for the pair of model plus harness: each vendor's native product is a different system. For your own evals: grade the behavior you can see, review a sample of the disagreements by hand, check that the verifier accepts different ways of doing the right thing. If you use a model as the judge, state how uncertain it is.
The rules that matter live where the action happens
In the WhatsApp agent I built for CVing, a recruiting platform, the agent can link a phone number to an account and submit applications. Two actions with real consequences, for real people. I decided to move the security checks out of the prompt and into the code that performs each action. The model can ask to link an account as often as it likes, in the most convincing tone in the world: if the verification isn't there in that conversation, the action doesn't run. The prompt explains the intended behavior, the code decides whether the effect is allowed. A beautifully written prompt is still an instruction, and instructions can be talked around. I follow the same order when an agent keeps repeating a mistake in a codebase: I fix the code first, then add a check, and write instructions last (why).
Same logic for identity. Who is writing on WhatsApp is something the channel tells you, and if the model tries to use a different number, the channel wins. In Yempik's Company Brain, which company an assistant acts on is decided by the person's sign-in. Whatever comes from the model is a request to authorize.
And stored data stays untrusted when it goes back in front of the model: in CVing even the names of files candidates upload get checked again when they're read back, because storing data doesn't make it more trustworthy. None of these steps makes you immune to prompt injection. They shrink the surface, nothing more.
Business state lives outside the conversation
In CVing an application starts from a link tied to a specific job opening. In long conversations, with lots of data to collect, at some point the model lost track of the first message and made up the opening. I widened the conversation memory, but the real fix was saving the active opening as a fact in the system, which the actions use instead of whatever the model remembers. A longer memory helps interpret the conversation, and business truth has to be written somewhere else.
Another time a candidate got stuck. Their verification had expired mid-conversation, but the agent kept believing it was valid, never asked for a new code, and kept telling them to check their email, where nothing was arriving. Since then a verification lasts for a set time and lapses the moment the system that issues it rejects it.
In the same piece of work I sorted every write with one question: if I repeat it, do I cause harm? If not, an error means retry. If the important action already succeeded and only the record of it fails, the candidate moves on and the system notes what went wrong. “Stop everything on any error” and “always retry” both sound rigorous, and both break the user's path.
Same memory, different instructions, different result
It happened with my own agent. In my second brain its ingestions are deep: it reads the whole source, keeps numbers, attributions and contradictions. On a platform used by a whole company, writing into a similar memory through a different door, it produced thin entries. Same model, same memory underneath. What changed was the instructions it received: on one side depth, reasoning, provenance and a clear idea of when the work is done; on the other an invitation to break everything into small facts.
The fix was bringing the same substantive rules to every entry point, with a final check, and how much meaning actually survives is still to be measured. If you expose the same memory from several places, compare the instructions and the definition of “done” on each one.
Search has to be able to say there's nothing
A ranking always produces a winner, even when every candidate is wrong. In the Company Brain, keyword search was so strict that naturally phrased questions found almost nothing. Without that corroboration, questions that had nothing to do with the Brain still came back with approved decisions on top, just because they were the least far off.
We made keyword search more forgiving and decided that ranking first isn't enough: a result has to clear a relevance bar measured on real data, and “I have nothing” is a valid answer. That bar has to be reset every time the way the system measures similarity changes. Refusal deserves the same care in evaluation as search itself.
The objection: soon the model will do it all by itself
The strongest objection to investing in the layer around the model: models improve so fast that the layer will become useless. But the model you use today is frozen. In a session it can reach a real discovery, and then that discovery isn't kept anywhere. On a call with another builder I put it this way: once you've had some kind of epiphany, there's no way to get it into its weights.
That's why I invest in context, and I call things by their name. When I say a company brain gets more useful, I mean approved knowledge, procedures, examples and evals that accumulate. The model stays the same. And even today swapping models isn't an afternoon job: everything around it has to be checked again.
From Monday
- List the rules you have to control yourself and use native tools underneath.
- Put the checks for every consequential action inside the code that performs it.
- Take identity and company from the sign-in. Whatever the model asks for needs authorizing.
- Keep business facts outside the conversation, and give every verification an expiry.
- Test search's ability to say “nothing”.
- Build evals on your own work and check the verifier by hand.
// two of the off-topic questions that now come back empty-handed: buying snacks and hamster food.