An eval measures performance. A release gate decides what you can claim.

Series: How agent systems get built and monetized. Post 1 of 5.

A draft describing our AI product said that when the system did not know something, it said so. I held the claim until we could support it. Our benchmark results showed correct refusal rates of roughly 79% and 86% on two runs against a hand-written set of twelve unanswerable questions. Twelve is below the sample size our own eval spec requires before acting on a measured difference. On those tests, the product still answered some questions its sources could not support.

Investigating the behavior exposed a concrete failure path. When retrieval found nothing, the system supplied ten recently compiled notes and instructed the model to answer anyway. That instruction gave it material to work with, but no reason to believe the material answered the question.

The immediate product decision was to remove the claim. Fixing the behavior and establishing the evidence needed to restore that claim were separate pieces of work. We could describe what the product did without promising a level of reliability the measurements did not support.

That distinction is why I use both an eval and a release gate. The benchmark measured the behavior. The gate required the claim to match the evidence. A measurement without a release decision could leave misleading copy intact. A review without measurement could approve a plausible sentence that was false.

Make the release decision inspectable

In my publishing workflow, drafts produced by agents require a written gate verdict before publication. There are three outcomes: pass, pass with notes, or changes. A materially revised draft goes through again. A verdict applies to the version that was reviewed; it does not carry forward automatically when the substance or voice changes.

The written record matters because a silent pass is indistinguishable from a skipped gate. A week later, I need to know which version was judged and what the reviewer concluded. My recollection is not enough.

The refusal claim is the one the gate itself stopped. Two others came from the same discipline applied earlier, and it is worth being exact about which is which. A groundedness score quoted to three decimal places had no eval artifact behind it, and nobody could point to the run that produced it, so it did not ship. A description of the product running on modest hardware did not survive the measured latency. Neither of those was a gate verdict on a draft. Both were checks made before a draft existed. A metric I cannot trace is a hallucination with a decimal point, wherever it gets caught.

Across the month covered by this draft, the gate produced twenty-two verdicts across fifteen distinct drafts. Three required changes; eight passed with notes. That means half the verdicts were qualified or rejected. It does not mean half the distinct drafts needed changes, because a draft can be reviewed more than once.

Those counts show the gate was being used. They do not establish how many problems it missed, or whether every objection was useful. The refusal claim is the more meaningful evidence: it shows a consequential statement the process prevented us from publishing.

Keep the measurement separate

The eval harness runs an LLM-judged benchmark against hand-authored questions with expected answers, scoring accuracy, completeness, citation, and faithfulness, with a separate groundedness judge. We also investigated scoring variability. Small-sample, single-run comparisons were noisy enough that we moved to a three-vote median and a larger sample before using differences to guide changes.

A twenty-three-test retrieval drift suite runs in CI to catch the regressions it covers. That is useful protection, but it does not establish the quality of every answer. The scored benchmark runs on demand, rather than automatically, and the latest scored results available when I prepared this draft were about seven weeks old.

That limits what I can say about the current product. A behavior fix and passing regression tests do not automatically establish a new benchmark score. Before restoring a reliability claim, I need evidence that covers the revised behavior and the conditions the claim describes. The gate makes that gap visible; it cannot fill it.

Product leadership owns the claim

For me, the product responsibility is deciding what we can credibly put in front of a customer while the engineering and measurement work continues. That includes narrowing a claim when the evidence supports only a narrower promise, and withholding it when there is no defensible basis for publishing it.

The refusal example required exactly that judgment. Removing the sentence was a decision we could make immediately. Establishing how reliably the product declines unsupported questions required more work. Keeping those decisions separate let us correct the copy without pretending we had already resolved the underlying quality question.

The reason to hold that line is not tidiness. A capability claim does not stay in the draft it was written for. It moves into the datasheet, the deck, the RFP response, and eventually the contract, and every one of those readers treats it as a commitment rather than an estimate. AI claims are more exposed than most, because a buyer can test one in an afternoon and frequently does.

There is a pricing consequence that follows. A capability you cannot evidence is a capability you cannot tier on, cannot meter, and cannot defend when a renewal conversation asks what exactly is being paid for. Packaging rests on claims that hold under examination, which means the decision to publish a claim and the decision about what it is worth are the same decision made twice.

Before publishing a material AI capability claim, I now ask three questions:

• What evidence supports it?
• Which product version and conditions does that evidence cover?
• What change would require us to validate it again?

A recorded verdict makes the decision inspectable. An eval supplies the evidence. Product leadership owns the decision to ship.