Nobody Budgeted for the Agent: Metering AI Work Across Departments

On October 30, Cognition announced an enterprise governance solution for Devin, built in collaboration with major customers in banking, retail and telecom. Read the release carefully and the scope is narrow and specific: it addresses funding and oversight of AI agent usage at scale, for organisations where thousands of engineers across separate business units are consuming agents that are priced by consumption. The phrase the coverage keeps landing on is the who-pays problem, which is a good phrase because it names the actual friction. The person using the agent is a developer with a ticket. The person accountable for the number is a manager two levels away in a different cost centre. Nothing in the tooling in between used to care.

The user of an agent and the budget owner are almost never the same person.

1. What was announced, and what the release does not say

The disclosed facts are these. The product targets large organisations where agents are consumed across disparate departments and thousands of engineering seats. It was developed with customers in banking, retail and telecommunications, industries with heavy regulatory requirements and complicated internal structures. It is described as providing transparent funding of AI initiatives and the ability to track who pays, with budgetary control over usage that is decentralised by nature. What the release does not contain is any mechanics: no unit of measurement, no policy model, no API, no statement about how a budget is enforced when it runs out. I am not going to guess at them. What I can do is describe the design that any such product has to have, because the underlying problem is the same whether you buy it or build it.

# Attribution has to exist at the call site, or it is a guess. Wrap the agent's
# entry point once, and let the run id travel through every model and tool call.
import contextvars, json, time, uuid
from dataclasses import dataclass, asdict, field

RUN = contextvars.ContextVar("agent_run")


@dataclass
class RunContext:
    run_id: str
    cost_centre: str          # the thing finance reports on
    project: str
    ticket: str
    model: str = ""
    input_tokens: int = 0
    cached_tokens: int = 0
    output_tokens: int = 0
    tool_calls: int = 0
    started: float = field(default_factory=time.time)
    outcome: str = "open"

    def event(self, kind: str, **fields):
        # One line per billable step. Append to your warehouse, not the vendor's.
        print(json.dumps({"run": asdict(self), "kind": kind, **fields}))


def start_run(cost_centre: str, project: str, ticket: str) -> RunContext:
    ctx = RunContext(str(uuid.uuid4()), cost_centre, project, ticket)
    RUN.set(ctx)
    return ctx


def current() -> RunContext:
    ctx = RUN.get(None)
    if ctx is None:
        # Untagged runs are the leading cause of unexplained spend. Fail loudly
        # in staging and tag them as "unattributed" in production.
        ctx = RunContext(str(uuid.uuid4()), "unattributed", "unknown", "unknown")
        RUN.set(ctx)
    return ctx

# The test that matters: start a run, let the agent call a tool, and assert that
# the tool's record carries the same run_id. If it does not, that work is
# billable and invisible, which is the worst combination.

2. Consumption pricing broke the seat-based budget

A software budget built on seats has one property that finance departments quietly rely on: it is known in advance. You buy a hundred licences, you know the number, and the variance is zero. Agent pricing does not work that way. Cognition meters Devin in ACUs, normalised units of agent compute; other vendors meter tokens, or minutes, or tasks. Whatever the unit is called, the spend scales with how much work gets delegated, and that is decided by whoever is closest to the problem, one ticket at a time. Multiply that by a few thousand engineers and you get a number nobody forecast, arriving in a currency nobody budgeted, attributed to no one. That is not a procurement failure. It is what happens when a variable-cost tool is deployed through a process designed for fixed-cost tools.

# The ledger. One row per run, event-sourced from the per-step events, closed
# daily. Small enough to read in a review, which is the point.
import json, sqlite3
from pathlib import Path

db = sqlite3.connect("agent_usage.db")
db.executescript(
    "CREATE TABLE IF NOT EXISTS runs ("
    "  run_id TEXT PRIMARY KEY,"
    "  cost_centre TEXT NOT NULL,"
    "  project TEXT NOT NULL,"
    "  ticket TEXT NOT NULL,"
    "  model TEXT,"
    "  day TEXT NOT NULL,"
    "  input_tokens INTEGER DEFAULT 0,"
    "  cached_tokens INTEGER DEFAULT 0,"
    "  output_tokens INTEGER DEFAULT 0,"
    "  tool_calls INTEGER DEFAULT 0,"
    "  outcome TEXT DEFAULT 'open'"
    ");"
    "CREATE INDEX IF NOT EXISTS runs_day ON runs(day, cost_centre);"
)

def ingest(path: str):
    with open(path, encoding="utf-8") as fh:
        for line in fh:
            ev = json.loads(line)
            r = ev["run"]
            db.execute(
              "INSERT INTO runs (run_id, cost_centre, project, ticket, model, day,"
              " input_tokens, cached_tokens, output_tokens, tool_calls, outcome)"
              " VALUES (?,?,?,?,?,?,?,?,?,?,?)"
              " ON CONFLICT(run_id) DO UPDATE SET"
              "   input_tokens = excluded.input_tokens,"
              "   cached_tokens = excluded.cached_tokens,"
              "   output_tokens = excluded.output_tokens,"
              "   tool_calls = excluded.tool_calls,"
              "   outcome = excluded.outcome",
              (r["run_id"], r["cost_centre"], r["project"], r["ticket"],
               r["model"], r.get("day", "1970-01-01"),
               r["input_tokens"], r["cached_tokens"], r["output_tokens"],
               r["tool_calls"], r["outcome"]))
    db.commit()

def monthly_statement(month: str, rate_in: float, rate_cached: float, rate_out: float):
    rows = db.execute("""
      SELECT cost_centre, SUM(input_tokens), SUM(cached_tokens),
             SUM(output_tokens), COUNT(*)
      FROM runs WHERE day LIKE ? GROUP BY cost_centre ORDER BY 2 DESC
    """, (month + "%",)).fetchall()
    for centre, tin, tcached, tout, runs in rows:
        billed = (tin - tcached) / 1e6 * rate_in + tcached / 1e6 * rate_cached \
                 + tout / 1e6 * rate_out
        print(f"{centre:<24} runs={runs:<6} ${billed:>10,.2f}")

# Every row here is a claim you may have to defend. Keep the raw events too, so
# the claim can be re-derived by somebody who does not trust your rollup.

3. Attribution is an engineering problem, and it starts at the call site

Every chargeback system reduces to one question: can you say, for this unit of spend, which tenant it belongs to. For an agent that means the attribution has to be present where the call is made, not reconstructed afterwards from logs that were never designed to carry it. The practical minimum is a run record with a stable id, a cost centre, a project and a ticket reference, propagated through every model call and every tool call the run makes — including the ones the agent decides to make on its own, which are exactly the ones that surprise the budget owner. Naming the unit matters too. Tokens are easy to measure and hard to explain to a manager. Human-equivalent hours are easy to explain and hard to measure honestly. Pick the one you can defend, and store the raw counters regardless so you can re-derive the other later.

# Reserve before the spend. The point is not to be strict, it is to make an
# over-budget state a product decision with a defined cheaper path.
import time
from dataclasses import dataclass


@dataclass
class Budget:
    cost_centre: str
    limit_usd: float
    spent_usd: float = 0.0
    reserved_usd: float = 0.0

    def available(self) -> float:
        return self.limit_usd - self.spent_usd - self.reserved_usd


BUDGETS = {"payments-platform": Budget("payments-platform", 4_000.0)}


def estimate(prompt_tokens: int, max_output_tokens: int, rate_in: float,
             rate_out: float, cached_ratio: float = 0.0) -> float:
    fresh = prompt_tokens * (1 - cached_ratio)
    return fresh / 1e6 * rate_in + max_output_tokens / 1e6 * rate_out


def admit(cost_centre: str, prompt_tokens: int, max_output_tokens: int):
    """Return the plan this run may use, or refuse with a defined fallback."""
    b = BUDGETS[cost_centre]
    want = estimate(prompt_tokens, max_output_tokens, 1.0, 4.0, cached_ratio=0.6)
    if want <= b.available():
        b.reserved_usd += want
        return {"tier": "standard", "reservation": want}

    # Degrade before denying: a cheaper model, a shorter completion, a later slot.
    cheap = estimate(prompt_tokens, 2_000, 0.15, 0.6, cached_ratio=0.6)
    if cheap <= b.available():
        b.reserved_usd += cheap
        return {"tier": "degraded", "reservation": cheap,
                "note": "budget low: cheaper model, 2k completion cap"}

    return {"tier": "queued", "reservation": 0.0,
            "note": "over budget: queued until period close or an approved override"}

# Two properties worth testing: concurrent admissions must not each receive the
# full remaining amount, and the refusal path must be a documented behaviour
# rather than whatever the exception handler happened to do.

4. A budget that reports is an invoice; a budget that blocks is a control

The failure mode of most first-generation agent cost tooling is that it is a dashboard. It tells you, on the third of the month, that a team has already spent its quarter. By then the choice has been made. What actually works is enforcement before the spend, in the request path: reserve against the run's expected usage, deny or degrade when the reservation cannot be satisfied, and make the degraded path a product decision rather than an error page — a cheaper model, a smaller context, a queue. Reservation also fixes a subtle accounting problem, because concurrent runs against the same remaining budget cannot each be granted the full amount. And keep a deliberate exception route, because a release-week lockdown that nobody can override is how governance tooling gets disabled by the second week.

# The number a manager will ask for: cost per unit of work that actually
# landed. Compute it from the ledger joined to your delivery data.
import csv, sqlite3

db = sqlite3.connect("agent_usage.db")
db.execute("CREATE TEMP TABLE merged AS SELECT 1")  # placeholder for clarity

def outcomes(path: str) -> dict:
    # ticket -> (merged_prs, swallowed) from your VCS export
    out = {}
    with open(path, newline="", encoding="utf-8") as fh:
        for row in csv.DictReader(fh):
            out[row["ticket"]] = (int(row.get("merged_prs") or 0),
                                 row.get("swallowed", "no") == "yes")
    return out

def report(vcs_csv: str, rate_in=1.0, rate_out=4.0, rate_cached=0.1):
    deliv = outcomes(vcs_csv)
    rows = db.execute("""
      SELECT ticket, SUM(input_tokens), SUM(cached_tokens), SUM(output_tokens),
             COUNT(*)
      FROM runs GROUP BY ticket
    """).fetchall()

    for ticket, tin, tcached, tout, runs in rows:
        cost = ((tin - tcached) / 1e6 * rate_in
                + tcached / 1e6 * rate_cached
                + tout / 1e6 * rate_out)
        prs, swallowed = deliv.get(ticket, (0, False))
        if prs:
            print(f"{ticket:<18} ${cost:>8,.2f}  prs={prs}  ${cost/prs:>8,.2f}/pr")
        elif swallowed:
            # Spend with no artefact is not waste — it is the honest cost of
            # exploration, and the only way to keep that line visible is to log it.
            print(f"{ticket:<18} ${cost:>8,.2f}  abandoned (keep, in a separate line)")

# Two rules that keep this honest: never divide by zero-merges, and never
# average abandoned work into the cost per PR. Both flatter the number and
# both get caught the first time somebody reads the raw rows.

5. Reconciliation is where governance earns or loses trust

Your ledger and the vendor's invoice will disagree, and the only question is whether you find out from your own close or from a controller asking why the number moved. Build the reconciliation as a monthly job: sum your usage events by cost centre, join them against the invoice lines, and report variance per team with a threshold that means something — three percent is not noise at team level, it usually means a metering gap. The two most common causes are runs that were never attributed because a new entry point bypassed the tagging layer, and retries that the vendor bills and your wrapper collapses into one event. Neither is exotic. Both are invisible unless somebody joins the two datasets on purpose.

#!/usr/bin/env python3
# Reconcile your ledger against the vendor invoice, monthly. Variance at team
# level is the signal; a single total that roughly matches is not.
import csv, sqlite3, sys
from collections import defaultdict

THRESHOLD = 0.03  # 3% at team level is not noise

db = sqlite3.connect("agent_usage.db")
ours = dict(db.execute("""
  SELECT cost_centre, SUM(input_tokens - cached_tokens) / 1e6 * 1.0
       + SUM(cached_tokens) / 1e6 * 0.1 + SUM(output_tokens) / 1e6 * 4.0
  FROM runs WHERE day LIKE ? GROUP BY cost_centre
""", (sys.argv[1] + "%",)).fetchall())

theirs = defaultdict(float)
with open(sys.argv[2], newline="", encoding="utf-8") as fh:
    for row in csv.DictReader(fh):
        theirs[row["cost_centre"]] += float(row["amount_usd"])

bad = 0
for centre in sorted(set(ours) | set(theirs)):
    a, b = ours.get(centre, 0.0), theirs.get(centre, 0.0)
    delta = (b - a) / a if a else float("inf")
    flag = "  CHECK" if abs(delta) > THRESHOLD else ""
    print(f"{centre:<24} ours={a:>9,.2f} invoice={b:>9,.2f} delta={delta:>7.1%}{flag}")
    bad += bool(flag)

raise SystemExit(1 if bad else 0)

# When a line is flagged, look at two things first: runs tagged 'unattributed'
# (a new entry point skipped the wrapper) and retry counts on your side versus
# billable attempts on theirs. Those two explain most first-pass variances.

6. Buy the product or build the thirty lines — but decide before the first invoice

If a vendor sells you attribution, enforcement and reporting as one product, and it works with your identity provider and your cost-centre hierarchy, that is a reasonable thing to buy. The part you should not outsource is the schema, because that is the artefact you keep when you change vendors: run id, cost centre, project, ticket, model, input tokens, cached tokens, output tokens, tool calls, duration, and the outcome. Store it in your own warehouse, not only in the vendor's console. Then the decision at renewal is an engineering decision rather than a migration project, and the question at month end — who paid for this — has an answer that does not depend on anyone's dashboard loading.