A Vending Machine Is a Better Agent Benchmark Than Another Chat Leaderboard

Why a business simulation beats another leaderboard

Planning over 30 days is a different skill from answering in 30 seconds. Orders, deliveries, sales and rent unfold across simulated time, so an agent that decides well on Monday and forgets on Tuesday gets caught. Six machines compete for one bank account, one depot and one eight-hour working day, which makes resource allocation observable rather than rhetorical. Different customers, changing offers and unreliable suppliers force adaptation. Notes and reminders exist so the agent can track unfinished work, and forgetting to use them shows up in the balance.

Every tool call advances simulated time, which is the design decision that makes the whole thing work. There is no free introspection: checking inventory costs the same minutes as changing a price, so a model that explores endlessly runs out of working day before it runs out of ideas.

The team's own summary of the ambition is worth quoting because it sets the right expectation: the goal is to make a company benchmarkable enough that AI can research how to run it better. Vending is the smallest business they could encode, not a claim that retail is solved.

# The cheapest useful thing you can do today: watch a reference operator run the
# business with no model, no API key, and no Docker. Requires Python 3.11+.

git clone https://github.com/ProsusAI/vending-bench.git
cd vending-bench
python3 scripts/local_run.py --seed 1

# Expected output for seed 1 with the default configuration:
#   days operated      30
#   tool calls         506
#   units sold         998
#   FINAL BALANCE      €3,302.96
#
# Read the caveat in the README before you quote that number: the scripted reference
# has privileged catalogue knowledge. It is a calibration reference, not an AI result.
# If your agent scores below it, you have not proven the agent is bad; you have
# proven the environment is measuring something your agent has not learned to use.

Running it in five minutes, and the caps that keep it honest

The reference operator runs with Python 3.11 and nothing else: no model, no API key, no Docker. For seed 1 with the default configuration it operates 30 days in 506 tool calls, sells 998 units and finishes at €3,302.96. The README is careful to say the scripted operator has privileged catalogue knowledge and is a calibration reference rather than an AI result, which matters because it sets the floor for what a competent run looks like.

Putting a model in charge needs uv and an authenticated Claude CLI, and every trial is capped at $5 API-equivalent and 1,800 seconds. Critically, the project states that these limits do not guarantee completion and an unfinished run scores zero. That single design choice removes the incentive that ruins long-horizon evals: an agent that burns its budget slowly and reports confidently.

Keep batches clean. One output directory per batch, one set of flags, one seed list. Most benchmark disputes I have seen are not about models, they are about somebody's second run quietly using a different flag set.

# Now put a model in charge. The runner needs uv and an authenticated Claude CLI,
# and every trial is capped so a loop cannot quietly spend your month.

uv run --with fastmcp==2.11.3 python scripts/run_models.py \
  --models haiku \
  --seeds 1 \
  --workers 1 \
  --budget 5 \
  --timeout 1800 \
  --output results/my-first-run

# Two caps, both load-bearing: $5 API-equivalent per run and a 1,800-second limit.
# The README is explicit that these do not guarantee completion -- an unfinished run
# scores zero. That is a feature. A benchmark where running out of budget still
# counts as a result teaches agents to be slow and confident.
#
# Use a fresh output directory for every batch. Mixing runs from different flags in
# one directory is how leaderboards end up comparing apples to a Docker image.

Completed is not competent, and the numbers show it

Five frontier models all completed the 30-day run, so completion tells you nothing about quality. What separates them is the economics. Opus 5 turned €1,500 into €4,497.21, roughly a 200% return, at a cost of $13.55. Sol turned it into €3,357.76, about 124%, for $1.91. Luna finished at €1,601.59 — barely above break-even — for eight cents. Haiku 4.5 lost €162.33.

Compute the ratio, not the rank. Sol captured 88% of Opus's profit for 14% of the API bill, which is the kind of sentence that changes a procurement decision. Haiku's cost was also small in absolute terms, yet it destroyed capital, and the correct reading is not that it is cheap; it is that the task exceeded it.

Two honest caveats. This is one simulation seed, and live model sampling does not establish significance; the project says exactly that. And the batch above ran under benchmark v2 rules, which the repository now marks superseded by v3 — a shorter working day, overnight banking, a 45-minute restock errand, a 30-minute item swap, and no per-tool-call levy charged to the business. The numbers are not comparable with a v3 run, and the maintainers have not rewritten them.

# The headline failure this benchmark is designed to expose: an agent can complete
# every day and still be a terrible shopkeeper. Bank balance alone hides that, because
# a high balance might mean good decisions, a better location, or a competitor that
# gave snacks away. So compute the economics per run, not just the ending cash.

def economics(final_balance, start=1500.00, api_cost=0.0, tool_calls=0):
    profit = final_balance - start
    return {
        "profit": round(profit, 2),
        "profit_per_tool_call": round(profit / tool_calls, 3) if tool_calls else None,
        "return_on_start_pct": round(profit / start * 100, 1),
        "eur_profit_per_usd_cost": round(profit / api_cost, 1) if api_cost else None,
    }

# Published 30-day batch, seed 1 (benchmark v2 rules -- see the next snippet):
#   Opus 5      4,497.21  cost 13.55
#   GPT-5.6 Sol 3,357.76  cost  1.91
#   Sonnet 5    3,289.12  cost  5.77
#   Luna        1,601.59  cost  0.0770
#   Haiku 4.5   1,337.67  cost  1.98
print(economics(4497.21, api_cost=13.55, tool_calls=506))
print(economics(1337.67, api_cost=1.98, tool_calls=506))

# Sol returning 88% of Opus's profit for 14% of the API bill is the number a
# procurement conversation actually needs. Rank by profit, then read the cost column.

Rank by profit, then read the API cost column. Sol took 88% of Opus's profit for 14% of the bill.

Building your own long-horizon eval without rebuilding the plumbing

The part worth stealing is the packaging. The vending world is a Harbor task, and Harbor — from the creators of Terminal-Bench — supplies container environments, agent adapters and trial-running infrastructure. The simulator follows Harbor's MCP sidecar pattern and exposes its 22 tools over the Model Context Protocol, so the benchmark is a piece of infrastructure rather than a bespoke script. Prosus says it built it in less than a week with that scaffolding.

That tells you what your own evaluation should look like: a container, a sidecar exposing tools over MCP, a start state, a sealed scoring step, and a results directory that records provenance. You do not need a novel harness for your domain; you need a domain, an economic rule that accumulates over time, and a score that cannot be gamed by the agent under test.

The security notes are part of the design rather than an afterthought, and they transfer directly. The simulator holds one world in one process with no authentication, so every trial must be isolated. There is no live score endpoint, and finalising a run seals it irreversibly before the score is returned, so an agent that reaches the verifier forfeits instead of gaining an advantage. Configuration is loaded from TOML through environment variables, which means those paths must point only at files you control.

# The trap in every public benchmark: comparing runs that were not comparable.
# This one changed rules between versions. v3 shortened the working day to 09:00-17:00,
# replaced cash boxes with overnight banking, made filling a machine a 45-minute errand,
# added a 30-minute swap_item, and stopped charging the business for tool calls.
# The published model batch predates all of that and is kept as a record, not a live
# leaderboard. Tag every result with its provenance and refuse to average across tags.

import json, hashlib, subprocess
from datetime import datetime, timezone

def provenance(extra=None):
    return {
        "captured_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "commit": subprocess.check_output(
            ["git", "-C", "vending-bench", "rev-parse", "HEAD"], text=True).strip(),
        "benchmark_version": "v3.0.0",     # read from the repo, do not trust memory
        "seed": 1,
        "harness": "harbor",
        **(extra or {}),
    }

def comparable(a, b):
    keys = ("benchmark_version", "seed", "harness", "commit")
    return all(a.get(k) == b.get(k) for k in keys)

run_a = provenance({"model": "kimi-k3.1-pro"})
run_b = provenance({"model": "your-incumbent"})
print(json.dumps([run_a, run_b], indent=2))
print("comparable:", comparable(run_a, run_b))

# If you change one flag, change the version tag too. A benchmark you cannot audit
# is a vibe with a table around it.

A checklist for treating it as an evaluation

Six checks before you quote a score. Isolate the simulator on a private network per trial. Hash the config TOML you used, because configuration is arbitrary-code-adjacent and a silent override invalidates the run. Record the benchmark version, the harness, the commit and the seed next to every result. Keep the $5 and 1,800-second caps in the command line rather than in a wiki. Keep API keys out of the results directory, since published trajectories contain raw model text. And decide in advance what happens to failed runs, because an unfinished run scoring zero is a design choice you should not casually override.

Then use the 365-day variant as a different question rather than a harder one. A month tests operation; a year tests whether the agent's strategy survives seasonality and drift. If your agent wins at 30 days and collapses at 365, that is the most useful sentence your evaluation can produce.

Finally, resist the urge to turn it into a leaderboard for your own team. The benchmark's real value is comparative experiments you can repeat: change one instruction, one memory scheme, or one tool, rerun, and see whether the bank balance moves. That is the whole argument for measuring agents on outcomes instead of vibes.

#!/usr/bin/env bash
# Pre-flight for running the benchmark as an evaluation rather than a demo.
set -euo pipefail

echo "== isolation =="
# The simulator is not multi-tenant: one world per process, no authentication,
# meant to run on a private network inside a single trial's Docker Compose project.
# Exposing it to a shared network is a deployment mistake, not a supported config.
docker network ls | grep -c vending || echo "no vending network yet (fine on first run)"

echo "== scoring integrity =="
# There is no live /state or /score endpoint. POST /verifier/finalize seals a run
# irreversibly before returning the score, so an agent that reaches it forfeits.
grep -R "finalize" vending-bench/tasks/vending-bench/environment 2>/dev/null | head -3

echo "== config provenance =="
# VENDING_CONFIG, VENDING_SET and other VENDING_* variables load and override a TOML
# file from disk. Only point them at files you control, and hash the config used.
sha256sum vending-bench/tasks/vending-bench/environment/sim-server/vending/config.toml 2>/dev/null \
  || shasum -a 256 vending-bench/tasks/vending-bench/environment/sim-server/vending/config.toml

echo "== budget and time caps =="
echo "per run: --budget 5 --timeout 1800; unfinished run scores zero"

echo "== keys =="
test -n "${OPENROUTER_API_KEY:-}" && echo "openrouter key present" || echo "set OPENROUTER_API_KEY in an env file you pass explicitly"

echo "== results dir =="
mkdir -p results/$(date +%Y%m%d)-$(git -C vending-bench rev-parse --short HEAD)

Every trial runs isolated: one world per process, no shared network, no live score endpoint.