你的 Agent 单测全绿，质量还是掉了：把评估回路跑起来

能跑通一个演示的 Agent，和能一直跑得好的 Agent，是两个不同的问题。前者一个周末就够，后者需要一条回路：写用例、跑、评分、读失败、改 Agent、再跑。变好就留下，变坏就回滚。Google 在 10 月 28 日公布了一整条回路的实操记录，用的是 Agents CLI v1.8.0 里的 eval 命令集。工具其实是里面最不重要的部分，值得花时间的是中间那两次实验——它们的平均分都很难看，而当时所有单测、类型检查、lint 规则全是绿的。这一类失败，靠写更干净的代码是躲不掉的。

4.00 分、标准差 1.41，比一片 5.0 的健康得多。

一、数据集才是产品，四个用例就能起步

脚手架自带的数据集只有一条无关痛痒的用例，对爬山式优化毫无用处——样本只有一个的时候，你根本看不见变化。这份实操把它扩到四条，分别探测不同行为：一句打招呼、一个 mock 工具能处理的城市天气、一个工具会走默认分支的城市天气，以及一个跑题问题，看 Agent 自己怎么处理。就四条，第一次运行的平均分是 4.0，标准差 1.41。这个离散度才是有用的信号。一片 5.0 只能说明两件事之一：要么 Agent 完美了（它不完美），要么数据集太简单，什么也测不出来。小数据集在这里是特性而不是偷懒：只有当回路便宜到每次改动都能跑，它才划算。

{
  "eval_cases": [
    {"name": "greeting",
     "agent_data": {"turns": [{"events": [{"author": "user",
       "content": {"parts": [{"text": "hello"}]}}]}]}},
    {"name": "sf_weather",
     "agent_data": {"turns": [{"events": [{"author": "user",
       "content": {"parts": [{"text": "What is the weather in San Francisco?"}]}}]}]}},
    {"name": "rome_weather",
     "agent_data": {"turns": [{"events": [{"author": "user",
       "content": {"parts": [{"text": "How is the weather in Rome today?"}]}}]}]}},
    {"name": "off_topic",
     "agent_data": {"turns": [{"events": [{"author": "user",
       "content": {"parts": [{"text": "Who won the 2024 World Series?"}]}}]}]}}
  ]
}

// tests/eval/datasets/basic-dataset.json from the walkthrough, verbatim in
// structure. Four cases is deliberate: this file has to be cheap enough to
// run on every commit, and wide enough that a single change cannot move all
// of them the same way.

二、所有既有检查都没抓住的那次回归

为了演示这条回路，他们制造了一个很真实的失误：一个同时在做好几个 Agent 项目的开发者，把错误的系统提示词粘进了天气 Agent，它变成了一个厨师人设，每次回复末尾都要推荐一道菜。代码编译得过，Agent 也照样回答，lint、单测、类型检查全绿——因为这些东西里没有任何一条知道「这个 Agent 应该用什么语气说话」。重跑 generate 加 grade，平均分从 4.00 掉到 2.75。诊断信息在逐用例的拆解里：打招呼从 5.0 掉到 4.0，因为它自说自话推了一道菜；sf_weather 从 4.0 掉到 2.0；跑题题从 5.0 掉到 3.0，因为答完之后又附了一道菜谱。一行提示词，三种不同的失败形态，而中间那个才是真正会上线的那种。

# tests/eval/eval_config.yaml — declare the metric the judge runs.
metrics_to_run:
  - custom_response_quality

custom_metrics:
  - name: custom_response_quality
    custom_function_file: response_quality.py

# response_quality.py grades each response 1-5 on accuracy, relevance and
# clarity, using the judge model. Keep the judge pinned: changing it mid-
# experiment invalidates every comparison you made before the change.

# Then, the two commands that make up one turn of the loop:
#
#   agents-cli eval generate \
#     --dataset tests/eval/datasets/basic-dataset.json \
#     -o tests/eval/output/
#
#   agents-cli eval grade \
#     --traces tests/eval/output/eval_dataset_*.json \
#     --config tests/eval/eval_config.yaml
#
# Baseline output from the walkthrough:
#   num_cases_total: 4  num_cases_valid: 4  num_cases_error: 0
#   mean_score: 4.0000  stdev_score: 1.4142

三、一定要读逐用例差异，否则你会修错地方

sf_weather 这条最值得盯。厨师人设把模型的注意力从已注册的 get_weather 工具上拽走了，于是 Agent 不再调用它，开始编天气数据。这没有任何单测能抓住：代码在跑，工具还注册着，schema 没变，变的是模型在这个工具周围的行为，只有把回答放在上下文里打分的端到端评估才看得见。Agent 的回归大多是这个形状——改动落在某个组件里，伤害却出现在完全别的地方，而且表现形式是一段听起来很合理的回答，而不是一个异常。平均分只告诉你「有东西动了」，逐用例清单才告诉你「哪几条动了」，而这几条就点名了是哪个组件变了。

# The mean is one line; the diagnosis is in the per-case diff. eval compare
# writes a structured JSON diff, so pull the per-case movement out of it.
import json
import sys


def cases(path):
    with open(path, encoding="utf-8") as fh:
        data = json.load(fh)
    out = {}
    for item in data.get("eval_case_results", []):
        name = item.get("name") or item.get("case_name")
        score = (item.get("metrics") or {}).get("custom_response_quality")
        if name and score is not None:
            out[name] = float(score)
    return out


baseline, candidate = cases(sys.argv[1]), cases(sys.argv[2])

print(f"{'case':<16}{'base':>7}{'cand':>7}{'delta':>8}")
for name in sorted(set(baseline) | set(candidate)):
    b = baseline.get(name, float("nan"))
    c = candidate.get(name, float("nan"))
    flag = "  <-- regression" if c < b else ""
    print(f"{name:<16}{b:>7.2f}{c:>7.2f}{c - b:>8.2f}{flag}")

# On the walkthrough's two experiments this prints:
#   chef prompt   greeting 5->4, sf_weather 4->2, rome_weather 2->2, off_topic 5->3
#   model swap    greeting 5->5, sf_weather 4->3, rome_weather 2->2, off_topic 5->1
# Same mean drop. Different work to do.

四、同样的平均分跌幅，完全不同的原因

第二个实验把模型后端从 gemini-3.8-flash 换成 gemini-2.5-flash，提示词恢复成正常的，平均分 2.75——跟厨师那次一模一样，这就是陷阱所在。逐用例看完全是另一回事：打招呼守住 5.0，sf_weather 只是在措辞上掉了一分，rome_weather 因为基线本来就在编数据所以停在 2.0，而跑题题从 5.0 崩到 1.0，原因是便宜模型以「这件事还没发生」为由拒绝了——这是训练截止时间造成的痕迹。只看平均分，你会判定这次换模型和粘错提示词一样糟；看用例，你会判定它对大多数负载是合理的取舍，只是对近期事件类问题确实有短板。两个前提得记住：换模型与提示词是互相作用的，正确比法是「为弱模型调过的提示词」对「为强模型调过的提示词」；评分模型必须保持不变，否则你测的是另一回事。

# A merge gate that reads the result file instead of a dashboard. Two rules:
# nothing may fall by more than one point, and the mean may not slip past a
# floor. Both are cheap, and both are arguments you only have to have once.
import json, sys

CASE_FLOOR = -1.0     # no single case may drop more than one point
MEAN_FLOOR = -0.30    # overall drift budget per change


def summary(path):
    with open(path, encoding="utf-8") as fh:
        data = json.load(fh)
    metrics = data.get("summary_metrics", {}) or {}
    mean = float(metrics.get("mean_score", 0.0))
    per_case = {}
    for item in data.get("eval_case_results", []):
        name = item.get("name") or item.get("case_name")
        score = (item.get("metrics") or {}).get("custom_response_quality")
        if name and score is not None:
            per_case[name] = float(score)
    return mean, per_case


base_mean, base_cases = summary(sys.argv[1])
new_mean, new_cases = summary(sys.argv[2])

failures = []
for name, base in base_cases.items():
    delta = new_cases.get(name, base) - base
    if delta < CASE_FLOOR:
        failures.append(f"{name}: {base:.2f} -> {new_cases[name]:.2f}")

if new_mean - base_mean < MEAN_FLOOR:
    failures.append(f"mean: {base_mean:.2f} -> {new_mean:.2f}")

if failures:
    print("eval regression, refusing to merge:")
    for line in failures:
        print("  " + line)
    raise SystemExit(1)

print(f"eval OK: mean {base_mean:.2f} -> {new_mean:.2f}")

# Keep thresholds in the repo, not in someone's head. The number you pick is
# less important than the fact that a number exists and someone else can see it.

五、把回路挂上定时，因为地面会自己动

Agents CLI 不自带调度器，也不需要。eval 子命令就是普通的退出码和文件，一个 cron 任务或者 Cloud Scheduler 触发器，每晚对一份稳定的回归数据集跑 generate 加 grade，就能抓住那些没人发公告的漂移：模型厂商悄悄升级、Agent 某个工具背后的数据变了、某个依赖在脚下挪了位置。最后这个不是假设——检索索引重建、第三方 API 改了默认返回结构，行为就会变，而你仓库里没有任何 diff 会显示它。夜里那次跑起来之后，门禁只多一步：读结果文件，跟上一次绿色的运行比，任一条用例跌幅超过你的阈值、或者平均分越过预算，就让任务失败。

#!/usr/bin/env bash
# Nightly drift check. No scheduler ships with the CLI; a five-line cron entry
# is enough, and it catches the change nobody announced.
set -euo pipefail
cd "$(dirname "$0")/.."

DATE=$(date +%Y%m%d)
OUT="artifacts/scheduled_runs/$DATE"
LAST_GREEN="artifacts/scheduled_runs/last_green.json"
mkdir -p "$OUT"

agents-cli eval generate \
  --dataset tests/eval/datasets/regression.json -o "$OUT/"
agents-cli eval grade \
  --traces "$OUT"/eval_dataset_*.json \
  --config tests/eval/eval_config.yaml \
  --output "$OUT/"

RESULT=$(ls -t "$OUT"/grade_results/*.json | head -1)

if [ -f "$LAST_GREEN" ]; then
  if ! python3 scripts/eval_gate.py "$LAST_GREEN" "$RESULT"; then
    # Whatever pages a human on call. The message matters less than the fact
    # that a regression at 03:00 is not discovered by a customer at 10:00.
    curl -sS -X POST "$ALERT_WEBHOOK" \
      -H 'Content-Type: application/json' \
      -d "{\"text\": \"agent eval regression on $DATE: $RESULT\"}" || true
    exit 1
  fi
fi

cp "$RESULT" "$LAST_GREEN"
echo "eval green for $DATE"

# The failures this catches most often are not your code: a vendor's silent
# model upgrade, a tool's data changing shape, a dependency bump. None of them
# produce a diff you can review, and all of them move the score.

六、如果要只记一句话

值得照抄的不是那个 CLI，而是这个习惯：把 Agent 质量当成「用一份自己的小数据集、按计划去量」的东西，而不是发版之后凭感觉推断的东西。每次跑两组数字就够起步了：平均分，和最差的那一条。等回路稳定了再加成本和 p95 延迟，因为「质量守住、价格减半」和「质量守住、价格不变」是两个完全不同的决策。还要留住那几条跑起来最便宜的用例，而不是报告里最好看的那几条——这条回路真正回本的时刻，是某个无聊的周三下午，一次依赖更新悄悄改变了你的 Agent 说的话。