Amazon Told Its Engineers the AI Usage Metrics Are Deeply Flawed. Then It Kept Using Them

Amazon published the memo behind its revised software development engineer role guideline on October 27, and the interesting part is not the new vocabulary. It is two paragraphs that contradict each other, written by people who knew they were contradicting themselves. Engineering leadership, including Distinguished and Senior Principal Engineers, named token usage and AI tool adoption among the input metrics teams are measured on, then wrote that these metrics are deeply flawed because they are disconnected from what matters, warned that using them as a target drives negative outcomes, and in the next breath said the numbers still correlate well with outputs and are appropriate to use with care. That is not sloppiness. That is an accurate description of the measurement problem every engineering organisation now has, and it is worth reading as a specification rather than a press release.

Amazon renamed the target: the Renaissance Developer, blending AI tools with human judgment.

1. What the memo actually says, in order

The structure matters. First, history: Amazon's role guidelines have deliberately avoided naming technology, because the company spans too many businesses, and that held through two decades of cloud, microservices and serverless. Second, the break: senior engineers debated whether this time is different and concluded it is, because AI is changing the work faster and more fundamentally than those earlier shifts. Third, the refusal to write it down: any definition of good AI-powered engineering written today will look antiquated by this time next year, and antiquated definitions freeze practice in place. Fourth, the pivot, quoting Bezos from 2012: stop asking what will change in five to ten years, ask what will not change. Fifth, the answer — impact on customers, businesses and teams — and with it the label, the Renaissance Developer, blending AI tools with human judgment and domain expertise.

# Cost per merged change. The output-side denominator that makes a token
# number mean something. Run monthly, keep every run.
import json, subprocess
from collections import defaultdict

def gh(path):
    out = subprocess.run(
        ["gh", "api", path, "--paginate"],
        capture_output=True, text=True, check=True,
    ).stdout
    # gh --paginate concatenates JSON arrays; join them before parsing.
    return json.loads("[" + out.replace("][", ",") + "]")

def merged_prs(repo, since):
    raw = gh(f"repos/{repo}/pulls?state=closed&per_page=100&sort=updated&direction=desc")
    return [p for p in raw if p.get("merged_at") and p["merged_at"] >= since]

def tokens_by_session(log_path):
    spend = defaultdict(int)
    with open(log_path, encoding="utf-8") as fh:
        for line in fh:
            if not line.strip():
                continue
            rec = json.loads(line)
            spend[rec.get("pr_number")] += int(rec.get("total_tokens", 0))
    return spend

REPO, SINCE = "your-org/your-repo", "2026-10-01"
prs = merged_prs(REPO, SINCE)
spend = tokens_by_session("agent-sessions.jsonl")

priced = [(p["number"], spend.get(p["number"], 0)) for p in prs]
billable = [(n, t) for n, t in priced if t > 0]
total = sum(t for _, t in billable)

print(f"merged PRs: {len(prs)}   with agent spend: {len(billable)}")
print(f"tokens per merged PR: {total / max(len(billable), 1):,.0f}")

# Publish this next to the token total, never instead of it. A token number
# with no denominator is a number that can only go up.

2. The two sentences that every engineering manager should steal

The memo does not just concede that token counts and lines of code are proxies. It names the failure mode. Tokenmaxxing, it says, is at odds with frugality. Measuring lines of code is at odds with invent and simplify. Both are Amazon leadership principles, which means the memo is describing a metric that actively fights a stated value rather than a metric that is merely imprecise. Then it does something rarer: it tells engineers that a negative result is a legitimate finding. If your work does not benefit from current AI tooling, that is a finding, not a failing — document why, share it with peers, bring it to your manager. An engineer who can explain where AI does not yet help their work is demonstrating the judgement the company is asking for. Read that alongside the pressure in the same memo to adopt AI, and you have the actual management posture of 2026: adopt, but bring evidence.

# Two metrics with opposite failure modes. Both can be gamed; together they
# are much harder to game at once.
from datetime import datetime, timedelta

def defect_escape_rate(merged_prs, incidents, window_days=14):
    if not merged_prs:
        return None
    cutoff = max(p["merged_at"] for p in merged_prs)
    linked = {i["pr"] for i in incidents if i.get("pr")}
    in_window = [
        p for p in merged_prs
        if datetime.fromisoformat(p["merged_at"].replace("Z", "+00:00"))
        >= datetime.fromisoformat(cutoff.replace("Z", "+00:00")) - timedelta(days=window_days)
    ]
    if not in_window:
        return None
    escaped = sum(1 for p in in_window if p["number"] in linked)
    return escaped / len(in_window)

# How to read the pair:
#   cost per merged PR down  +  escape rate flat    -> real efficiency
#   cost per merged PR down  +  escape rate up      -> you are buying speed
#                                                      with quality; name it
#                                                      out loud before the
#                                                      customer does
#   cost per merged PR up    +  escape rate down    -> possibly a tougher
#                                                      problem, check scope
#
# Never report one of the two alone. A single number always becomes a target;
# that is what targets are for.

3. The percentile advice is the most operational line in the document

Amazon recommends looking at the 10th and 90th percentile of AI adoption rather than the mean. The tenth percentile is where teams are not adopting effectively, and the instruction is to understand why rather than to send a reminder. The ninetieth percentile is where teams use AI heavily, and the instruction is to check whether the delivery increase actually arrived. Both directions catch a real failure that averages hide: a team stuck at 5% adoption and a team at 95% adoption with flat throughput produce the same mean as two teams at 50%, and only one of those three scenarios is healthy. The memo also ties this to proximity: engineers and managers who are close to their product and customers get the most out of agentic development, and Amazon says it wants to reverse the drift of ICs away from customers that came with scale. Nothing about that requires a token dashboard.

# Lines of code is the metric the memo calls out by name. Here is what it
# actually counts, which is why it is easy to inflate and useless for sizing.
import subprocess

def loc_by_language(path="."):
    out = subprocess.run(
        ["git", "ls-files", "-z"], cwd=path,
        capture_output=True, text=True, check=True,
    ).stdout.split("\0")
    files = [f for f in out if f]

    counts = {}
    tracked = 0
    for f in files:
        try:
            with open(f, encoding="utf-8") as fh:
                n = sum(1 for _ in fh)
        except (UnicodeDecodeError, OSError):
            continue                      # binaries, locks, images
        ext = f.rsplit(".", 1)[-1] if "." in f else "(none)"
        counts[ext] = counts.get(ext, 0) + n
        tracked += n

    for ext, n in sorted(counts.items(), key=lambda kv: -kv[1])[:10]:
        print(f"{ext:<12} {n:>8,}")
    print(f"{'TOTAL':<12} {tracked:>8,}")
    return tracked

# Why the number is meaningless for performance review:
#   reformatting, generated clients, lockfiles and vendored dependencies all
#   inflate it; deleting 400 lines of duplicate logic shrinks it. The change
#   most worth celebrating in your repo this quarter may show up as -400.
#
# Size a change instead by files touched, hunks, and the review surface:
#   git diff --stat <base>...<head>
#   git diff --numstat <base>...<head> | wc -l

4. Why input metrics survive anyway, and what to do about it

The honest reason token counts and adoption rates stay in the room is latency. Outcome metrics — did the customer get value, did the change reduce cost — arrive months late, are confounded by everything else happening in the business, and cannot be attributed to one team. Input metrics arrive tomorrow. Amazon's compromise is to use them with care and judgement while treating them as instrumentation rather than as a scoreboard, and to keep looking for better metrics. The engineering response to that is not to refuse the input metric but to make it expensive to game. If your team is measured on tokens consumed, publish cost per merged change next to it. If your team is measured on lines changed, publish defect escape rate next to it. Any single number becomes a target; a pair with opposite failure modes is much harder to game in both directions at once.

# Percentile review, not average review. This is the one operational habit
# from the memo that is worth copying this week.
import json, statistics

TEAMS = json.load(open("team_ai_adoption.json", encoding="utf-8"))
# each entry: {"team": "checkout", "adoption": 0.71, "throughput_delta": 0.14}

adoption = sorted(TEAMS, key=lambda t: t["adoption"])

def pct(rows, p):
    if not rows:
        return None
    idx = max(0, min(len(rows) - 1, round((len(rows) - 1) * p)))
    return rows[idx]

p10, p90 = pct(adoption, 0.10), pct(adoption, 0.90)
median = statistics.median(t["adoption"] for t in TEAMS)

print(f"median adoption: {median:.0%}")
print(f"p10: {p10['team']} at {p10['adoption']:.0%}  -> ask why, do not send a reminder")
print(f"p90: {p90['team']} at {p90['adoption']:.0%}  -> throughput delta {p90['throughput_delta']:+.0%}")

# The high-adoption team with a near-zero throughput delta is the more
# interesting conversation. Heavy tool use that does not move delivery is
# either a workflow problem or a measurement problem. Both need a human.

5. The Renaissance Developer is a workload claim, not a compliment

The label is easy to read as flattery. Read the paragraph around it instead: the memo expects engineers to blend AI tools with human judgment and domain expertise, leaders to have used these tools on production work and to be able to describe one task where they helped and one where they did not, and managers to connect inputs to outputs inside their own team. That is a wider job description than the one it replaces, delivered at the same headcount. The saving grace is the emphasis on judgement over throughput: the memo explicitly says the leaders having the most impact are the ones doubling down on learning, and that teams which followed curiosity while balancing delivery and standards saw the most positive impact from AI adoption. Curiosity is not a metric. That is exactly why it is the right thing to reward.

# "A finding, not a failing": keep a ledger of tasks where AI did not help.
# Cheap to write, and in a year it is the most useful document you have.
import json
from datetime import date

LEDGER = "ai-does-not-help.jsonl"

def record(task, category, why, tried):
    row = {
        "date": date.today().isoformat(),
        "task": task,
        "category": category,       # e.g. "legacy build", "perf tuning"
        "why_ai_did_not_help": why,
        "what_was_tried": tried,
    }
    with open(LEDGER, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(row, ensure_ascii=False) + "\n")

record(
    task="Make the 2019 Gradle build reproducible on a clean machine",
    category="legacy build",
    why="The failure is environment drift across three undocumented JDK patches; "
        "the model keeps proposing dependency bumps that are already applied",
    tried="Full-file context, then targeted excerpts, then a written repro script",
)

# Review it quarterly, in the same meeting where you review adoption.
# Two outcomes: either the tooling gap closes (and you delete rows), or the
# category becomes an argument for buying different tooling. Both are wins,
# and neither happens if nobody writes the ledger down.

6. A measurement stack that survives the contradiction

Build three layers and label them honestly. Layer one, instrumentation that answers tomorrow: model calls, tokens, cost per change, agent-started share, review comments per hundred lines. Nobody is evaluated on these; they exist to explain anomalies. Layer two, the pair metrics that resist gaming: cost per merged change next to defect escape rate, cycle time next to revert rate, review depth next to merge speed. Layer three, the outcome a human owns: did this change move the number the customer cares about. Then run the percentile review quarterly, not weekly, because adoption behaviour moves slowly and a weekly view of a slow signal produces panic, not insight. And write down the tasks where AI did not help. That document is worth more in a year than any dashboard you build this quarter.