Clean evals vs dirty reality

A recent benchmark reported a result that is hard to ignore.

The strongest model passed 66.50% of the tasks on its first attempt. When the benchmark asked for reliability across twenty attempts, the result fell to 47.53%.

Same model. Same tasks. Different question.

The first score asks whether the agent can do the work. The second asks whether I can depend on it.

We often talk as if those are interchangeable. They are not.

A clean evaluation removes distractions so we can compare models. A production system adds them back: incomplete responses, incorrect data, stale state, tool failures, retries, and the occasional answer that looks finished but is missing the part that mattered.

That creates two reasonable camps.

Team Clean Evals wants controlled conditions. Hold the harness steady, define the task precisely, remove infrastructure noise, and measure the capability of the model. If the environment changes between runs, the score stops telling us what improved.

Team Dirty Reality wants the opposite. Break the tool. Return half the data. Leave residue from a previous run. Make the service slow. If the agent only works when every dependency behaves, the benchmark may be accurate and still useless for deployment.

I think we need both. But not for the same decision.

A clean benchmark is an instrument. Its job is to isolate a signal.

If I want to know whether one model plans better than another, I do not want a random API timeout deciding the winner. If I want to test whether a new prompt improves tool selection, I need the rest of the environment to stay still. Controlled conditions make failures easier to reproduce and improvements easier to attribute.

Without that discipline, every result becomes an argument about the test. Did the model fail, or did the browser stall? Was the plan wrong, or did the tool return malformed JSON? Did the new version regress, or did one dependency have a bad afternoon?

Clean evaluations also make engineering work possible. A team needs a small set of deterministic cases it can run after changing a prompt, tool description, model, or policy. Those tests should be quick enough to run often and stable enough to catch a real regression.

That is the strongest case for Team Clean Evals. It is not leaderboard obsession. It is the need to change one thing and learn something from the result.

But isolation has a cost. The more successfully a benchmark removes noise, the less evidence it provides about what happens when the noise returns.

Team Dirty Reality has its own easy failure mode.

Adding random failures does not make an evaluation useful. It can just make it noisy.

A test that occasionally kills the network, changes the task, corrupts a file, and then produces one combined score tells me very little. I cannot tell whether the agent misunderstood the goal, selected the wrong tool, retried badly, or encountered a scenario no system could recover from.

Realistic evaluation still needs structure.

DuMateBench, released in August, is interesting because it does not treat "real world" as a mood. It evaluates 200 tasks in environments that include insufficient resources, unstable tool or network behavior, and noisy workspaces. The point is not to make the benchmark harder for sport. It is to preserve conditions that normal evaluations often erase.

That is the version of dirty reality I find useful: named perturbations with observable consequences.

Return an incomplete record and see whether the agent notices the missing fields. Make a service time out and see whether it retries, degrades, or loops. Leave an old artifact in the workspace and see whether it mistakes residue for current state. Give two tools slightly different answers and see whether it invents certainty.

Each disturbance should answer a specific question about the system. Otherwise we have replaced a clean measurement with a dramatic demo.

The failure I care about most is not a timeout. A timeout is honest.

It is a partial or incorrect answer that looks complete.

Imagine an agent preparing a risk summary from three systems. The first returns the full record. The second returns stale data without marking it stale. The third returns only the first page because pagination failed.

The agent produces a polished summary. Every sentence is plausible. No tool call threw an error. The run ends cleanly.

The answer is wrong.

A normal task-completion check may pass it. The agent wrote the requested document, used the expected tools, and emitted valid output. The real failure lives in the state of the evidence. One source was old and another was incomplete.

This is why "the agent finished" and "the work is correct" need separate measurements.

Thinkingbox makes a related point by grading stateful business workflows through executable checks of the final state. Its pass@1 and pass^20 gap exposes something a single successful run cannot: capability is not reliability repeated over time.

I do not need an agent that never gets things wrong. I need one that knows when its answer stopped being trustworthy, or a system around it that notices.

That requirement changes the evaluation.

The obvious response is to ask the agent to verify its own work.

That helps. It should inspect whether every source responded, whether required fields are present, whether timestamps are current, and whether the final claims are supported by the retrieved evidence. These are cheap checks and many of them can be deterministic.

But self-checking has a limit. The same agent may carry the same mistaken assumption into the review. Asking it "Are you sure?" often produces a more confident explanation of the original answer.

A second agent can help, but only if it has a real job.

For normal work, I would use a verifier. It reviews the answer, the evidence, and the execution trace. It asks whether the conclusion follows from what the first agent actually saw. It can reject an answer with missing support or send it back for another pass.

For critical work, review is not enough. A second agent should solve the task independently, ideally without inheriting the first agent's reasoning. Then compare the outputs, evidence, and final state. Agreement does not prove correctness, especially when both agents share a model or source. Disagreement is still valuable because it tells us not to proceed quietly.

The verification cost should follow the consequence.

A low-risk internal summary may need schema checks and one verifier. A migration, payment, security change, or public claim may justify an independent run and a human escalation when the results diverge.

Using two agents for everything would be expensive theater. Using one agent for every critical decision would be cheap until the first silent failure.

I would keep the clean benchmark. Then I would add a deployment evaluation with explicit disturbances.

For each important workflow, I would write down:

I would report the clean capability score and the disturbed reliability score separately. Combining them hides the decision.

The clean score helps choose and improve the model. The dirty score helps decide where it can act, what controls it needs, and whether anyone should trust it unattended.

The Double Measurement Confound in Agent Benchmarks calls out a related problem: benchmark scores can mix model capability with choices made by the surrounding harness and evaluation stack. That is another reason to publish more than one number. A production result belongs to the whole system.

I understand why teams lead with pass@1. It is simple. It fits in a release post. It tells a clean story about progress.

Production does not run once.

It runs on the incomplete response, the stale record, the fifth retry, and the day when two dependencies disagree. It runs often enough that the unlikely failure eventually becomes somebody's ordinary Tuesday.

So I would not ask only, "Can the agent do this?"

I would ask, "What happens when the evidence is wrong, and does the system know?"

A clean benchmark measures capability. A dirty one measures whether you can sleep.

The next debate is Skills vs Memory: what should an agent learn about a person, and what should live in an explicit procedure instead?