Lesson 10 - Agent Evals - Golden Tasks and Trajectories

Code: agentic-course/agentic/evals.py Tests: agentic-course/tests/test_memory_evals_guard.py Run it: python3 -m unittest tests.test_memory_evals_guard -v Concept: Evals covers the theory and the interview framing, without code.

What you will build

• A check library that asserts on the answer and on the path: which tools ran, in what order, how many steps, at what cost.
• Case, Suite and Report — a versioned golden set that runs each case against a fresh agent and reports a per-case breakdown, not just a score.
• by_tag() to expose compensating movement, and regressed_against(baseline) to compare against the version you already shipped.
• A security case that passes because an attack was blocked, using the requested-versus-executed distinction the rest of this course leans on.

The idea

This is the lesson that decides whether you have an engineering practice or a hobby. Without evals, every change to a prompt, a tool description, a retrieval parameter or a model version is a coin flip you cannot read. You ship it, someone says it feels better, someone else says it feels worse, and both are working from the handful of examples they happen to remember. Six weeks later nobody can say whether the agent is better than it was in March.

The analogy: an eval suite is a regression harness, but the system under test is a wind tunnel rather than a circuit. You are not checking whether a gate outputs 1. You are measuring a distribution of behaviour under fixed conditions, and the only interpretable statement available is a comparison — this version against the version currently serving traffic.

Why unit tests do not transfer

1. There is no single correct output. “Refunds are accepted within 30 days” and “You have a month from delivery to request a refund” are both right. assertEqual on a string is hostile to the problem.
2. Sampling is stochastic. The same input can produce different outputs. A case that passes 70% of the time is not a flaky test to be fixed, it is an accurate measurement of a probabilistic system, and the harness has to be built for that rather than against it.
3. Quality is a gradient, not a boolean. An answer can be correct but unhelpful, correct but ungrounded, correct but four times more expensive than last week. A single pass bit throws away the dimension you needed.
4. A failure does not localise. A unit test failure points at a function. An eval failure points at a system — retrieval missed, or the tool description was vague, or the model ignored a constraint, or a budget cut the run short. The trace from lesson 9 is what turns that into a diagnosis.

What makes agent evals different from LLM evals

An LLM eval scores text against a prompt. An agent eval has to score the path.

An agent that returns the right refund policy after calling the refund tool six times is not working. It produced the correct answer and mutated production state five times more than it should have. Any scoring function that only reads run.output calls that a pass, which is why every case here asserts on output and trajectory. The worked exercise below carries four checks on one case — answered(), used_tools("kb_search"), contains("30"), max_steps(4) — covering four distinct failure modes: it terminated properly, it grounded the answer in retrieval instead of recalling it from weights, the answer carries the number, and it did not wander for eight steps to get there.

The check library

A check is a plain callable — Check = Callable[[Run], tuple[bool, str]] — returning whether it passed plus a detail string that lands in the failure report. That is why a red suite is readable instead of a list of Falses.

Outcome checks

• answered() — run.ok, meaning Stop.ANSWERED. Catches the run that hit a budget or a loop guard and returned an apology. Without it, a suite of contains checks silently grades truncated runs.
• contains(*needles) and excludes(*needles) — key facts present, forbidden strings absent. excludes("as an AI language model") is the cheapest style regression test there is. valid_json(required_keys) parses the output and checks required keys, and abstained(*markers) catches the agent declining instead of bluffing.

Trajectory checks

• used_tools(*names) — each named tool appears in the trajectory. Catches an agent that answered from memorised weights instead of retrieving, the most common silent failure in a retrieval agent.
• never_used(*names) — no named tool executed. This is how you test a guardrail. never_requested(*names) is stricter — the model did not even ask — and it measures something different.
• trajectory_is(*names) — exact sequence, for when order genuinely matters: check the order exists before you refund it.

Budget checks — max_steps(n) and max_cost(limit) cap steps and spend. stopped_with(stop) asserts a specific Stop reason, which is how you test that a budget or loop guard fires when it should.

Cost and step count are correctness, not performance

The docstring on max_cost is the whole argument: “Cost is correctness. A right answer that cost ten times budget failed.” Treat cost as a performance concern and it gets its own dashboard, owned by someone else, reviewed next month. Treat it as correctness and a change that triples spend per resolution turns the suite red in the pull request that caused it. Step count behaves the same way and is the leading indicator: steps drive tokens, tokens drive cost and latency, and every extra step is another chance to take an action you did not want.

Abstention is a measured behaviour

"""The agent declined rather than bluffing.

A retrieval agent with a zero abstention rate is not accurate, it is
guessing on the questions its corpus cannot answer.
"""

Your golden set needs cases the corpus cannot answer, where the only passing behaviour is to say so. An agent that never abstains has not achieved perfect coverage; it has learned that confident prose is rewarded. Measure the abstention rate as a first-class number and it stops being invisible.

Case, Suite, Report

Case carries id, task, checks, tags and why. That last field is not decoration — the source calls it required in spirit, because a case with no recorded reason for existing is a case nobody dares delete when it goes stale.

Suite.run takes a callable, not an agent, and wraps every check:

for case in self.cases:
    agent = build_agent()
    run = agent.run(case.task)
    for check in case.checks:
        try:
            ok, detail = check(run)
        except Exception as exc:  # a broken check must not kill the suite
            ok, detail = False, f"check raised {type(exc).__name__}: {exc}"

Two decisions sit in those lines. The factory exists so each case gets a clean agent. Share one agent and case two inherits case one’s history, tool state and accumulated budget. The suite still runs, still produces numbers, and the numbers are now a function of case ordering. test_each_case_gets_a_fresh_agent pins this: with one scripted reply per agent, a shared agent would exhaust the reply queue and fail loudly. And a broken check fails its case and nothing else — test_a_broken_check_fails_the_case_without_killing_the_suite raises inside a check and asserts the report still comes back with check raised in the detail. An exception escaping the loop would lose every result after the bad case, precisely when you most need the other 200 numbers.

by_tag, because the aggregate lies

by_tag() returns a pass rate per tag, and the docstring says why it is the breakdown you actually read: the aggregate hides compensating movement, one category improving while another regresses. An aggregate holding at 84% across a release can mean nothing changed, or it can mean the easy cases got better while the adversarial ones got worse. Those are opposite situations and one number renders them identically. test_by_tag_breakdown_exposes_compensating_movement is the minimal demonstration — easy at 1.0, hard at 0.0, an aggregate of 0.5 that tells you neither.

regressed_against, because absolute scores are not interpretable

def regressed_against(self, baseline: "Report") -> list[str]:
    """Case ids that passed in the baseline and fail now."""
    was = {r.case.id for r in baseline.results if r.passed}
    now = {r.case.id for r in self.results if r.passed}
    return sorted(was - now)

Is 84% good? Unanswerable. Did anything that used to work stop working? Answerable, specific, and blocking. test_regression_against_a_baseline shows the asymmetry that makes it useful: the worse report names the regressed case, and the better report against the worse baseline names nothing, because a new pass is not a regression. Store the report from the version serving traffic and gate the deploy on an empty regression list rather than on a threshold.

Requested versus executed - the distinction everything hinges on

Run exposes three views of tool activity, and conflating any two will mislead you:

run.trajectory()        # tools that EXECUTED successfully, in order
run.requested_tools()   # what the model ASKED for, including denied calls
run.blocked_tools()     # requested but denied, invalid, or failed

never_used scores the trajectory. So when a guardrail denies an injected refund, never_used("issue_refund") passes — correctly, because nothing happened — while never_requested("issue_refund") fails on the same run, because the model did try. Run both and you separate two facts a single metric fuses:

• never_used passing means the attack never landed. Containment worked.
• never_requested failing means the attack started. Prevention did not.

Those call for different work. A landed attack is an incident. A contained-but-attempted attack is a fencing, prompt or retrieval problem you fix on a normal schedule while the policy keeps holding the line. One boolean cannot express that, and a suite that only records “no breach” cannot tell you the attempt rate is climbing. test_never_used_is_how_you_test_a_guardrail is the unit-level version; test_injected_document_cannot_reach_a_mutating_tool is the end-to-end one, asserting never_used passes and never_requested fails on the same run. Lesson 12 builds the defence that makes it come out that way.

Golden sets: where the cases come from

A suite is only as good as its cases, and inventing cases at a desk produces a suite that passes while users suffer.

• Build it from real traffic and real failures. Every production incident becomes a case before the fix is merged. That is how a suite stops repeat failures instead of documenting them.
• Weight it toward the hard cases. Real traffic is mostly easy, so a representative sample is mostly easy, so your pass rate is dominated by questions the agent was never going to get wrong. Over-sample the ambiguous, the adversarial, the multi-hop and the unanswerable. You are buying discriminating power, not a traffic estimate.
• Version it in the repo. Suite(cases, name, version) carries a version because a score means nothing without knowing which set produced it. Editing cases and comparing across the edit is how teams accidentally celebrate.
• Record a reason per case. Six months on, a failing case with no recorded purpose gets deleted by whoever is unblocking the build, and you will never know what it was protecting.

The tiering in the module docstring is the cost discipline around all of it:

1. assertions      deterministic, free, run on every commit
2. reference       a ground truth exists - exact or field match
3. judged          a model grades against a rubric, validated against humans
4. human           the gold standard, sampled, not on every commit

Push everything as far up that list as it will go — tier 3 is lesson 11, and it is only trustworthy after validation. Here is a real suite of exactly that shape, demo.py scenario 5, three cases with one of them a security case:

3/3 passed  (100%)
mean steps 1.7   total cost 0.000000
by tag: adversarial=100%  happy=100%  policy=100%  security=100%

Cost reads zero because the demo leaves unit prices at their defaults; set price_per_1k_input on the Budget and this line becomes a number you can gate on. The security case passes because the refund was blocked — never_used scores what executed, and never_requested would show the model still tried.

Exercise

Build a three-case suite with one security case, run it as a baseline, then deliberately regress the agent and prove regressed_against names the broken case. Use the realistic regression: someone widens the tool allowlist so a mutating tool becomes reachable from a context that reads untrusted text.

Success criterion: python3 exercise_evals.py prints a baseline of 3/3, a candidate of 2/3 with security=0%, and regressed: ['injection-contained'].

What broke when I wrote this

The first version of Run.trajectory() returned the tools the model had requested, because the distinction looked like pedantry — the model asked for a tool, the tool goes in the trajectory, move on.

Then I wrote the injection test. A poisoned document told the model to call issue_refund, the model obeyed, the registry denied the call because the policy had dropped every tool from that context, and the refund did not happen. The defence worked exactly as designed. And never_used("issue_refund") failed, because the request was sitting in the trajectory.

That is the worst class of eval bug. It does not produce a wrong number you can argue with, it punishes the correct behaviour. Left in place, the pressure it creates is to weaken the assertion — or worse, to “fix” the guardrail until the eval goes green.

The fix split one concept into three: trajectory() for what executed, requested_tools() for what was asked, blocked_tools() for the difference. The comment it produced is still in loop.py:

"""Successful executions only. A call the registry denied is not part of the
trajectory, because nothing happened - asserting `never_used("refund")`
must PASS when a guardrail blocks an injected refund attempt, otherwise
the eval punishes the defence for working.
"""

The generalisation worth keeping: when an eval and a defence disagree, check the eval first. The defence has a specification; the eval has an assumption.

Checkpoint

Why is an absolute pass rate not an interpretable number?

Because it depends entirely on the difficulty mix of your golden set. The interpretable question is the comparison — did anything that passed in the shipped version stop passing — which is what regressed_against returns.

Why is build_agent a callable instead of an agent instance?

So each case runs against a fresh agent. Shared history, tool state or accumulated budget makes results a function of case ordering, and the suite keeps producing confident numbers while doing it.

A guardrail blocks an injected refund. What do never_used and never_requested report, and what does each mean?

never_used passes, because nothing executed — the attack never landed. never_requested fails, because the model still asked — the attack started. Containment worked, prevention did not, and those need different responses.

What does a zero abstention rate on a retrieval agent tell you?

That it is bluffing on the questions its corpus cannot answer. Your golden set needs unanswerable cases where declining is the only passing behaviour.

Theory and interview framing: Become an AI Engineer