The evaluation suite ran against the agent's plan for faster account recovery and reported a pass. Every case in the suite was about links: an expired link gets no session, a reused link gets no session, a valid link gets one. The plan satisfied all three. Nobody had written a case about accounts that hold a payment method, so the suite had no opinion about them, and the plan went to build with a green result beside it.

The rule it should have met was already settled: an account holding a payment method requires step-up verification before account recovery can issue a new session. The new recovery path omitted it. The unit tests passed, the eval passed, and the omission reached the reviewer with two green marks in front of it.

This is a fictional scenario, carried through several articles in this series from ticket to dashboard. It describes no real system, team, incident, deployment, or measurement. It is here because it separates two things that get merged the moment an agent starts writing code: what a passing check tells you, and what it cannot.

Why More Tests Would Not Have Caught It

Your first guess is more coverage: more unit tests, more eval cases, a stronger model behind the agent. None of those touch the failure. A test checks the code against an expectation someone encoded. An eval checks the agent's behavior against a case someone wrote. The fixture's expectations and cases were complete for everything that had been written down, and the rule about payment methods had not been written down anywhere a check could see. Widening coverage widens what is checked. It does not widen what is known.

In the fixture, the recovery test file held two tests: a valid link issues a session, and an expired or reused link does not. Both passed, and both were correct. The third test, the one that says a payment-method account is sent to step-up verification instead of being issued a session, did not exist, because the acceptance criterion it would encode did not exist. You cannot cover a criterion that nobody wrote.

The rest of this article is about keeping that argument useful. At every handoff in a delivery system that admits agents, four things are fixed in advance: what the agent may do, what artifact persists, who decides, and what evidence permits progression. This article is about the last of those, and about the honest limits of the answer. It is a recommendation for how to think about checks, not a policy; your organization's security policy, repository rules, release process, and incident runbooks take precedence wherever they disagree. Every vendor reference in it was last reviewed on 2026-09-02.

Evidence, Timing, and Decisions Are Three Different Axes

Teams tend to keep one list of "checks" that mixes three different kinds of thing. A unit test, an eval, a CI job, a code review, and a release approval end up as five items in one column, and the column hides the fact that they answer different questions.

The first axis is evidence mechanisms, the things that produce evidence. There are four. Deterministic tests check that the code does what someone encoded. Offline evals check an agent's behavior against explicit cases and criteria, without the agent acting on anything real. Runtime checks evaluate the agent's behavior while it acts, such as a policy check before a tool call is allowed through. Production monitoring watches what shipped rather than what was expected.

The second axis is execution points: when a mechanism runs. Locally, in CI on every change, on a schedule, or at runtime. This is a timing, not a kind of check. A test that runs in CI tells you what the same test tells you locally, and an eval that runs weekly tells you the same thing it would tell you on every change, only less often. Putting "CI" in the list of checks is the most common way the column goes wrong.

The third axis is accountable decisions, which are made by people and are not checks at all. A product or domain owner reviews intent against the product. A code owner reviews implementation against risk. A named release authority authorizes the irreversible or production action. Evidence feeds these decisions. It does not make them.

The diagram is a conceptual representation of how the axes connect for one change. Read it top to bottom, and notice the dashed node at the end.

The dashed node is the fixture. Every mechanism on the first axis takes its expectations from the top of the chain. A constraint that never entered the intent record, the acceptance criteria, the repository's context, a policy, or an evaluation set has no path into any of them, at any execution point, and no amount of running them recovers it. The sources agree on the shape without using this vocabulary. IBM's page on the lifecycle for building agents, reviewed on 2026-09-02, separates offline evaluation during build and CI from in-the-loop evaluation at runtime and describes both as checks against predefined benchmarks, policies, and expected behavior. Salesforce's guide to the same lifecycle treats agent-specific testing and session tracing as stages of their own. Both are describing the agent as the product. The distinction holds when the agent is a participant in delivering something else.

What a Passing Test Tells You

The sample below is a tested example. It is the fixture's recovery decision reduced to a pure function and the test that encodes the missing criterion, written in JavaScript so it can run in this repository with Node's built-in test runner and no dependencies; the logic is the same in any language. decideRecovery is the runnable counterpart to the plan's RecoveryService.decide(account, link). This sample starts after link validation: link.valid is the precomputed result, and an expired or reused link must arrive as false. Read the second branch of the function, and ask which test would not have existed in the fixture.

// recovery-decision.mjs
// Fictional recovery decision from the series fixture. Not production code.
export function decideRecovery(account, link) {
  if (!link.valid) return 'Reject'
  if (account.holdsPaymentMethod) return 'RequireStepUp' // C1, encoded as AC2
  return 'IssueSession' // AC1
}

// recovery-decision.test.mjs
import { test } from 'node:test'
import assert from 'node:assert/strict'
import { decideRecovery } from './recovery-decision.mjs'
 
const validLink = { valid: true }
const expiredLink = { valid: false }
 
test('AC2: a payment-method account is sent to step-up, not issued a session', () => {
  assert.equal(decideRecovery({ holdsPaymentMethod: true }, validLink), 'RequireStepUp')
})
 
test('AC1: an account without a payment method is issued a session', () => {
  assert.equal(decideRecovery({ holdsPaymentMethod: false }, validLink), 'IssueSession')
})
 
test('AC3: an expired or reused link gets neither a session nor a step-up prompt', () => {
  assert.equal(decideRecovery({ holdsPaymentMethod: true }, expiredLink), 'Reject')
})

Run with node --test recovery-decision.test.mjs under Node 24, all three tests pass. That pass tells you one thing: for the inputs the tests construct, the function returns what the tests expect. At this decision layer, expired and reused links form one prevalidated invalid-link class, and the example exercises one member of it. The pass does not establish that link validation maps both conditions to valid: false, that holdsPaymentMethod is populated correctly, that a controller delegates to this function, or that AC2 is the right criterion. Those require validator tests, an integration or architecture check, and an owner decision respectively.

Now delete the first test and the second branch, which is what the fixture's file looked like. The remaining two tests still pass, and they are still correct. A green result from that file is not wrong. It is silent about the one thing that mattered, and silence looks identical to success in a pull request.

What a Passing Eval Tells You, and What One Run Cannot

An eval is the same idea applied to the agent's behavior instead of the code's. The Claude Academy course's continuous-evals lesson, reviewed on 2026-09-02, defines an eval case as a real task with an expected or accepted outcome and explicit checks, and notes that some teams run their suite offline on a cadence rather than on every change. This series borrows only that shape without treating the vendor's implementation as a standard: a task, an expected outcome, and a check.

The case below is a configuration example in a vendor-neutral shape; no product reads this file. It is the eval the fixture did not have. The expected outcome is about the plan, not the code.

# eval-recovery-step-up.yaml (configuration example; vendor-neutral shape)
id: eval-recovery-step-up
task: >
  Given the ticket "faster account recovery", the intent record, and the
  repository, produce an implementation plan.
inputs:
  intent_record: intent-faster-account-recovery.md   # carries C1
  repository: recovery-service before the change
expected_outcome:
  - the plan names StepUpPolicy as an affected surface
  - the plan lists a test mapping a payment-method account to RequireStepUp (AC2)
  - the plan keeps the decision in RecoveryService, not in the controller
checks:
  - type: assertion
    rule: plan text mentions StepUpPolicy
  - type: assertion
    rule: plan lists a test for AC2
  - type: rubric
    rule: plan keeps the decision in RecoveryService rather than the controller
runs: 5
threshold: all runs pass
execution:
  - in CI, on every agent-produced plan for this service
  - on a schedule, against the current model and prompt

A pass here tells you this: on this run, with this model and this prompt, given an intent record that carries C1, the agent produced a plan that met the written outcome. Read the first input again. The case can only exist because the intent record carries the constraint. In the fixture it did not, so this case could not have been written, and the cases that were written were satisfied by a plan that omitted the rule. The fixture's eval passed that way, and it passed honestly.

One run is one sample. The agent is not deterministic, so a case that passed once can fail on the next run with nothing changed, and a case that failed once may be a rare path rather than a regression. Reading an eval result therefore needs two things the test result did not: a declared number of runs and a declared threshold, both set by the team that owns the case. The values in the example are placeholders. How to build a harness that runs, scores, and reports this is its own subject and not this article's.

Where a check runs changes how often you learn something, not what you can learn. Locally, the developer gets the fastest loop. In CI on every change, a regression is caught before merge. On a schedule, drift in the model or the prompt is caught even when no change was merged. At runtime, an in-loop check can stop an action before it happens, which is the one execution point that changes an outcome rather than reporting one. Production monitoring is the last mechanism and the only one that sees what actually shipped: in the fixture, the AC4 log of every recovery decision with its account class is where sessions issued to payment-method accounts would finally have become visible, after the fact.

Who Owns the Criteria

The two rows below are quoted from the ownership table this series uses, with their wording unchanged. The columns are what the agent does, what persists, who decides, and what evidence permits progression.

The rows have different owners, but the fixture failed before either row could help. A code owner can judge whether the test evidence supports the implementation and whether the technical risk is acceptable. A product or domain owner can judge whether the eval cases still describe recorded intent. Because C1 never reached the intent record or acceptance criteria, neither row contained evidence about it. Ownership can keep encoded criteria current; it cannot recover a constraint absent from the artifact chain.

The evidence that should persist follows from the rows. Test results tied to the specific change, not a dashboard aggregate. Eval results with the case identifiers, the run count, and the threshold, so a reviewer can see which cases were exercised and which were not. And one more record that the table implies: the date the cases were last confirmed against current intent, with the owner's name beside it. On a team with no formal owner for eval cases, the lighter equivalent is a single question in the change request: which acceptance criteria changed since these cases were written?

Where Evals Are Too Much

Ceremony scales with how open-ended the reasoning is and how expensive a plausible wrong result would be. Most changes sit low on both, and for them an eval is the wrong tool. A change to deterministic code needs a test, because the code's behavior is what varies; an eval of the agent that wrote it adds nothing the test did not already say. A routine patch update inside an approved dependency policy needs neither, beyond the suite that already exists. Eval cases earn their cost only where the agent's judgement is the thing that varies from run to run: plans, designs, triage, and any task where two correct answers can differ.

On a small team the whole apparatus can be one file of tests and one file of cases, owned by the same person who wrote the intent. That is fine. Two things should survive the simplification: the case that encodes a constraint has to exist before the agent's work is judged against it, and someone has to be able to say when the cases were last checked against what the product now requires.

The sources in this article define evals and say where they run. None of them shows that broader eval coverage predicts fewer failures of the fixture's kind, and this article does not claim it. A green suite is evidence about what was written. It is not evidence about what was not.

What No Eval Suite Can Tell You

I do not have a clean rule for how many runs and what threshold make an eval result trustworthy, and I distrust any number given without the case it applies to. What I am sure of is the order of operations the fixture got wrong. Write the criterion before the case, write the case before the agent's work is judged by it, and put a name beside the cases so someone is asked, at review time, whether they still describe the product. Everything a passing check can tell you starts with what somebody chose to encode.