Your Agent Passed Its Unit Tests and Still Got Worse: Running the Eval Loop

Building an agent that works in a demo and building one that keeps working are different problems. The first takes a weekend. The second takes a loop: write cases, run them, grade them, read what failed, change the agent, run again. Up if better, revert if worse. Google published a full walkthrough of that loop on October 28 using the eval namespace in Agents CLI v1.8.0, and the tooling is the least interesting part of it. What is worth your time is the two experiments in the middle, because both of them produce a bad mean score with every unit test, type check and lint rule still green. You cannot clean-code your way out of that class of failure.

4.00 out of 5 with a standard deviation of 1.41 is a healthier baseline than a flat 5.0.

1. The dataset is the product, and four cases can be enough

The scaffold ships with a dataset containing a single trivial case, which is useless for a hill-climb: you cannot see variation in a sample of one. The walkthrough expands it to four cases that probe different behaviours — a greeting, a weather question the mock tool handles, a weather question for a city the tool defaults on, and an off-topic question the agent has to decide how to handle. Four cases, and the first run returned a mean of 4.0 with a standard deviation of 1.41. That spread is the useful signal. A flat 5.0 tells you either the agent is perfect, which it is not, or the dataset is too easy to show you anything. Small datasets are a feature here, not a shortcut: the loop only pays off if it is cheap enough to run on every change.

{
  "eval_cases": [
    {"name": "greeting",
     "agent_data": {"turns": [{"events": [{"author": "user",
       "content": {"parts": [{"text": "hello"}]}}]}]}},
    {"name": "sf_weather",
     "agent_data": {"turns": [{"events": [{"author": "user",
       "content": {"parts": [{"text": "What is the weather in San Francisco?"}]}}]}]}},
    {"name": "rome_weather",
     "agent_data": {"turns": [{"events": [{"author": "user",
       "content": {"parts": [{"text": "How is the weather in Rome today?"}]}}]}]}},
    {"name": "off_topic",
     "agent_data": {"turns": [{"events": [{"author": "user",
       "content": {"parts": [{"text": "Who won the 2024 World Series?"}]}}]}]}}
  ]
}

// tests/eval/datasets/basic-dataset.json from the walkthrough, verbatim in
// structure. Four cases is deliberate: this file has to be cheap enough to
// run on every commit, and wide enough that a single change cannot move all
// of them the same way.

2. The regression that every existing check missed

To demonstrate the loop they introduce a realistic mistake: a developer working across several agent projects pastes the wrong system prompt into the weather agent, and it becomes a chef persona that ends every reply with a recipe suggestion. The code compiles. The agent still answers. Lint, unit tests and type checks all pass, because nothing in them knows what the agent is supposed to sound like. Re-running generate plus grade gave a mean of 2.75, down from 4.00. The per-case breakdown is where the diagnosis lives: the greeting dropped from 5.0 to 4.0 for an unsolicited recipe pitch, sf_weather fell from 4.0 to 2.0, and off_topic went from 5.0 to 3.0 by answering the question and then tacking on a recipe. One line of prompt, three different failure modes, and the middle one is the one that would have reached production.

# tests/eval/eval_config.yaml — declare the metric the judge runs.
metrics_to_run:
  - custom_response_quality

custom_metrics:
  - name: custom_response_quality
    custom_function_file: response_quality.py

# response_quality.py grades each response 1-5 on accuracy, relevance and
# clarity, using the judge model. Keep the judge pinned: changing it mid-
# experiment invalidates every comparison you made before the change.

# Then, the two commands that make up one turn of the loop:
#
#   agents-cli eval generate \
#     --dataset tests/eval/datasets/basic-dataset.json \
#     -o tests/eval/output/
#
#   agents-cli eval grade \
#     --traces tests/eval/output/eval_dataset_*.json \
#     --config tests/eval/eval_config.yaml
#
# Baseline output from the walkthrough:
#   num_cases_total: 4  num_cases_valid: 4  num_cases_error: 0
#   mean_score: 4.0000  stdev_score: 1.4142

3. Read the per-case diff, or you will fix the wrong thing

The sf_weather case is the one worth staring at. The chef persona pulled the model's attention away from the registered get_weather tool, so the agent stopped calling it and started inventing weather values. No unit test catches that: the code runs, the tool is still registered, the schema is unchanged. The model's behaviour around the tool shifted, and only an end-to-end eval that grades the response in context can see it. This is the shape of most agent regressions — a change lands in one component and the damage shows up somewhere else entirely, expressed as a plausible-sounding answer instead of an exception. The mean tells you that something moved. The per-case list tells you which cases moved, and those cases name the component that changed.

# The mean is one line; the diagnosis is in the per-case diff. eval compare
# writes a structured JSON diff, so pull the per-case movement out of it.
import json
import sys


def cases(path):
    with open(path, encoding="utf-8") as fh:
        data = json.load(fh)
    out = {}
    for item in data.get("eval_case_results", []):
        name = item.get("name") or item.get("case_name")
        score = (item.get("metrics") or {}).get("custom_response_quality")
        if name and score is not None:
            out[name] = float(score)
    return out


baseline, candidate = cases(sys.argv[1]), cases(sys.argv[2])

print(f"{'case':<16}{'base':>7}{'cand':>7}{'delta':>8}")
for name in sorted(set(baseline) | set(candidate)):
    b = baseline.get(name, float("nan"))
    c = candidate.get(name, float("nan"))
    flag = "  <-- regression" if c < b else ""
    print(f"{name:<16}{b:>7.2f}{c:>7.2f}{c - b:>8.2f}{flag}")

# On the walkthrough's two experiments this prints:
#   chef prompt   greeting 5->4, sf_weather 4->2, rome_weather 2->2, off_topic 5->3
#   model swap    greeting 5->5, sf_weather 4->3, rome_weather 2->2, off_topic 5->1
# Same mean drop. Different work to do.

4. The same mean drop, a completely different cause

The second experiment swaps the model backend from gemini-3.8-flash down to gemini-2.5-flash, with the helpful prompt restored, and gets a mean of 2.75 — identical to the chef regression, which is exactly the trap. Per case it is a different story: greeting held at 5.0, sf_weather lost one point on phrasing, rome_weather stayed at 2.0 where it had already been hallucinating, and off_topic collapsed from 5.0 to 1.0 because the cheaper model refused on the grounds that the event “has not happened yet”, a training-cutoff artefact. Read the mean alone and you would call the swap as bad as pasting the wrong prompt. Read the cases and you would call it a reasonable trade for most of the workload, with a real limitation on questions about recent events. Two caveats are worth keeping: model swaps interact with prompts, so compare a re-tuned weak model against a tuned strong one, and the judge model must stay constant or you are measuring something else.

# A merge gate that reads the result file instead of a dashboard. Two rules:
# nothing may fall by more than one point, and the mean may not slip past a
# floor. Both are cheap, and both are arguments you only have to have once.
import json, sys

CASE_FLOOR = -1.0     # no single case may drop more than one point
MEAN_FLOOR = -0.30    # overall drift budget per change


def summary(path):
    with open(path, encoding="utf-8") as fh:
        data = json.load(fh)
    metrics = data.get("summary_metrics", {}) or {}
    mean = float(metrics.get("mean_score", 0.0))
    per_case = {}
    for item in data.get("eval_case_results", []):
        name = item.get("name") or item.get("case_name")
        score = (item.get("metrics") or {}).get("custom_response_quality")
        if name and score is not None:
            per_case[name] = float(score)
    return mean, per_case


base_mean, base_cases = summary(sys.argv[1])
new_mean, new_cases = summary(sys.argv[2])

failures = []
for name, base in base_cases.items():
    delta = new_cases.get(name, base) - base
    if delta < CASE_FLOOR:
        failures.append(f"{name}: {base:.2f} -> {new_cases[name]:.2f}")

if new_mean - base_mean < MEAN_FLOOR:
    failures.append(f"mean: {base_mean:.2f} -> {new_mean:.2f}")

if failures:
    print("eval regression, refusing to merge:")
    for line in failures:
        print("  " + line)
    raise SystemExit(1)

print(f"eval OK: mean {base_mean:.2f} -> {new_mean:.2f}")

# Keep thresholds in the repo, not in someone's head. The number you pick is
# less important than the fact that a number exists and someone else can see it.

5. Put the loop on a schedule, because the ground moves without you

Agents CLI does not ship a scheduler and does not need to. The eval subcommands are ordinary exit codes and files, so a cron job or a Cloud Scheduler trigger that runs generate plus grade nightly against a stable regression dataset catches the drift nobody announced: the model vendor's quiet upgrade, a change in the data behind one of the agent's tools, or a dependency shifting under its feet. That last one is not hypothetical — behaviour changes when a retriever's index is rebuilt or a third-party API changes its default response shape, and no diff in your repository shows it. Once the nightly run exists, the gate is one more step: read the result file, compare against the last green run, and fail the job when any case drops by more than your threshold or the mean slips past a budget.

#!/usr/bin/env bash
# Nightly drift check. No scheduler ships with the CLI; a five-line cron entry
# is enough, and it catches the change nobody announced.
set -euo pipefail
cd "$(dirname "$0")/.."

DATE=$(date +%Y%m%d)
OUT="artifacts/scheduled_runs/$DATE"
LAST_GREEN="artifacts/scheduled_runs/last_green.json"
mkdir -p "$OUT"

agents-cli eval generate \
  --dataset tests/eval/datasets/regression.json -o "$OUT/"
agents-cli eval grade \
  --traces "$OUT"/eval_dataset_*.json \
  --config tests/eval/eval_config.yaml \
  --output "$OUT/"

RESULT=$(ls -t "$OUT"/grade_results/*.json | head -1)

if [ -f "$LAST_GREEN" ]; then
  if ! python3 scripts/eval_gate.py "$LAST_GREEN" "$RESULT"; then
    # Whatever pages a human on call. The message matters less than the fact
    # that a regression at 03:00 is not discovered by a customer at 10:00.
    curl -sS -X POST "$ALERT_WEBHOOK" \
      -H 'Content-Type: application/json' \
      -d "{\"text\": \"agent eval regression on $DATE: $RESULT\"}" || true
    exit 1
  fi
fi

cp "$RESULT" "$LAST_GREEN"
echo "eval green for $DATE"

# The failures this catches most often are not your code: a vendor's silent
# model upgrade, a tool's data changing shape, a dependency bump. None of them
# produce a diff you can review, and all of them move the score.

6. What to keep from this, in one line

The habit worth copying is not the CLI. It is treating agent quality as something you measure on a schedule with a small dataset you own, instead of something you infer from vibes after a release. Two numbers per run are enough to start: the mean, and the worst case. Add cost and p95 latency when the loop is stable, because a swap that holds quality at half the price is a different decision than a swap that holds it at the same price. And keep the four cases that are cheapest to run, not the ones that look most impressive in a report — the loop earns its keep on the boring Wednesday afternoon when a dependency update quietly changes what your agent says.