ReviewBench: GitHub's 219-PR Code Review Benchmark and the 96.6% Agreement Behind It

A demo set is not a benchmark

GitHub measured the real distribution of pull requests across language, repository size and change shape, and then sampled the benchmark to match it. Language and repo size mirror GitHub directly. Pull request size is deliberately reweighted toward the reviewable middle and tail, which removes the flood of one-line changes where review quality is trivially perfect and puts more weight on multi-file diffs where review actually matters.

The result is 219 pull requests from 187 public open source repositories spanning 19 languages, with the complete dataset published. That is small enough to run repeatedly and large enough that a single lucky or unlucky PR does not swing the score. The reweighting decision is also an admission that a perfectly representative sample is the wrong target when the question is review quality.

# Grounded precision and recall from a findings file: the strict, apples-to-apples pair.
import json
from collections import Counter

def grounded(findings, gold):
    '''findings: list of {id, pr_id}; gold: set of (pr_id, gold_id) tuples'''
    matched = {(f["pr_id"], f.get("gold_id")) for f in findings if f.get("gold_id")}
    found = matched & gold
    precision = len(found) / max(1, len(findings))
    recall = len(found) / max(1, len(gold))
    f1 = 2 * precision * recall / max(1e-9, precision + recall)
    return {"matched": len(found), "findings": len(findings), "gold": len(gold),
            "precision": round(precision, 3), "recall": round(recall, 3), "f1": round(f1, 3)}

GOLD = {("pr_17", "g1"), ("pr_17", "g2"), ("pr_42", "g3"), ("pr_88", "g4")}
FINDINGS = [
    {"pr_id": "pr_17", "gold_id": "g1"},
    {"pr_id": "pr_17", "gold_id": "g2"},
    {"pr_id": "pr_42", "gold_id": "g3"},
    {"pr_id": "pr_42", "gold_id": None},   # a real issue nobody in the gold set found
    {"pr_id": "pr_88", "gold_id": None},   # noise
]
print(grounded(FINDINGS, GOLD))   # recall 0.75, and the unmatched two decide augmented metrics

Ground truth from four sources, one rubric

No single producer finds everything worth finding in a diff. GitHub gathers candidate findings from real human reviewers, from issues inferred from an author's own follow-up commits, from deterministic analysis tools, and from multiple frontier models across model families. Then it semantically deduplicates overlapping findings so that agreement between producers does not inflate the golden set, and so the set does not inherit any one producer's blind spots.

The next step is what makes this trustworthy. The source of a finding does not determine whether it counts. A finding is a true positive only if it is true, relevant and non-trivial, judged under a single explicit rubric that is published along with the judge prompt and model configuration. That is a higher bar than treating any human comment as gold.

Before release, senior engineers who had not participated in building the dataset independently re-labelled every ground-truth finding from scratch. Their true and false positive judgments matched ReviewBench 96.6% of the time. Version pinning of dataset, judge and matcher means a result stays reproducible.

# The golden set is deduplicated semantically, because four producers often spot one bug.
# A cheap stand-in: cluster findings by (file, overlapping line range) then by similarity.
def dedupe(findings, similarity=0.85):
    clusters = []
    for f in sorted(findings, key=lambda x: (x["file"], x["start"])):
        for c in clusters:
            same_file = c[0]["file"] == f["file"]
            overlap = min(c[0]["end"], f["end"]) - max(c[0]["start"], f["start"]) >= 0
            if same_file and overlap:
                c.append(f)
                break
        else:
            clusters.append([f])
    merged = []
    for c in clusters:
        sources = sorted({x["source"] for x in c})
        merged.append({"file": c[0]["file"], "line": c[0]["start"],
                       "sources": sources, "votes": len(sources),
                       "text": max((x["text"] for x in c), key=len)})
    return merged

RAW = [
    {"file": "app/auth.py", "start": 88, "end": 92, "source": "human", "text": "Token not verified before use"},
    {"file": "app/auth.py", "start": 89, "end": 95, "source": "static", "text": "Unverified JWT payload"},
    {"file": "app/auth.py", "start": 90, "end": 91, "source": "llm",    "text": "You should verify the signature"},
    {"file": "app/db.py",   "start": 12, "end": 14, "source": "llm",    "text": "N+1 query in loop"},
]
for row in dedupe(RAW):
    print(row["votes"], row["file"], row["line"], row["text"])

Six metrics instead of two

Most benchmarks report precision and recall against a fixed golden set. ReviewBench reports two families. Grounded precision, recall and F1 use only existing gold labels: of the issues we know about, how many did you find, and what share of your findings matched one. Augmented precision, recall and F1 also score findings that match nothing in the golden set, with the judge deciding independently whether those unmatched findings are true or false positives.

That second family exists because a fixed golden set becomes incomplete the moment a system gets good. If your reviewer finds a genuine bug that no human, tool or model in the gold set surfaced, a grounded-only metric calls it noise. Augmented metrics let it count. The trade-off is that augmented recall expands the denominator based on what each agent discovers, so GitHub uses grounded recall as the headline for cross-system comparison and augmented metrics as a per-system diagnostic.

# Six metrics, and the beta that decides whether you rank by noise or by coverage.
def fbeta(precision, recall, beta=1.0):
    b2 = beta * beta
    return (1 + b2) * precision * recall / max(1e-9, b2 * precision + recall)

AGENTS = {
    "reviewer_a": {"precision": 0.71, "recall": 0.44},
    "reviewer_b": {"precision": 0.52, "recall": 0.63},
    "reviewer_c": {"precision": 0.83, "recall": 0.31},
}
for beta in (0.5, 1.0, 2.0):
    ranked = sorted(AGENTS.items(), key=lambda kv: fbeta(kv[1]["precision"], kv[1]["recall"], beta), reverse=True)
    print(f"beta={beta}:", [(name, round(fbeta(v['precision'], v['recall'], beta), 3)) for name, v in ranked])

# beta < 1 favours precision (fewer nits), beta > 1 favours recall (broader coverage).
# ReviewBench re-ranks the leaderboard when this preference changes, which is the point.

Every finding gets a severity and a category, so the leaderboard can be re-ranked by what you care about.

Severity and category slices decide the ranking

Every finding is labelled Critical, Medium or Low, and by category: correctness, security, reliability, maintainability, testing and more. Results can be sliced, and the leaderboard re-ranks when you change the precision-recall preference through the beta term in F-beta. Lower beta favours precision, higher beta favours recall.

This matters because there is no universal best reviewer. A team drowning in comments wants precision. A team shipping to regulated customers wants recall on security findings even at the cost of noise. A single number hides that.

# Severity slices carry information that the aggregate hides.
SLICES = {
    "critical": {"findings": 12, "matched": 11},
    "medium":   {"findings": 48, "matched": 31},
    "low":      {"findings": 96, "matched": 27},
}
total_findings = sum(v["findings"] for v in SLICES.values())
total_matched = sum(v["matched"] for v in SLICES.values())

for name, v in SLICES.items():
    share_of_hits = v["matched"] / total_matched
    share_of_noise = (v["findings"] - v["matched"]) / (total_findings - total_matched)
    print(f"{name:9} matched {v['matched']:3}  hit-share {share_of_hits:5.1%}  noise-share {share_of_noise:5.1%}")

# A reviewer whose hit-share on critical issues sits near its noise-share is not
# prioritising anything -- it is just producing volume.

The experiment that makes it credible

Benchmarks earn trust by predicting things. GitHub used ReviewBench to evaluate Copilot code review, and describes a lite-tier experiment where a multi-model ensemble combined several independent runs into one review. ReviewBench predicted higher precision, recall and comment volume, plus lower cost per review.

The online A/B test moved the same way: addressed rate, the online counterpart to precision, rose 8.0%, recall rose 13.6%, comment volume rose 61%, and cost per review fell 8.0%, all against control. Severity-level evaluation predicted a 227% increase in critical comments against 262% measured online. The magnitude was not perfectly predicted and does not need to be. Direction plus rough scale is enough to decide what earns an A/B test.

Keep that in mind when you read any vendor's benchmark claim, including this one. Offline scores are a filter for what to test in production, not a substitute for testing it.

# Offline movement is only useful if it tracks online movement. Keep both columns.
import statistics

EXPERIMENTS = [
    # name, offline_predicted_delta, offline_actual, online_delta
    ("ensemble_review", {"precision": +0.05, "recall": +0.10, "volume": +0.60, "cost": -0.05},
                        {"precision": +0.06, "recall": +0.12, "volume": +0.55, "cost": -0.07},
                        {"addressed_rate": +0.08, "recall": +0.136, "volume": +0.61, "cost": -0.08}),
]

def sign_agreement(predicted, observed):
    '''Do offline and online at least point the same way?'''
    pairs = [("precision", "addressed_rate"), ("recall", "recall"), ("volume", "volume"), ("cost", "cost")]
    agree = sum(1 for off, on in pairs if (predicted[off] > 0) == (observed[on] > 0))
    return {"agree": agree, "of": len(pairs), "ratio": agree / len(pairs)}

for name, pred, actual, online in EXPERIMENTS:
    print(name, sign_agreement(pred, online))
    drift = {k: round(actual[k] - pred[k], 3) for k in pred}
    print("  offline error:", drift)

# Published Copilot result for reference: addressed rate +8.0%, recall +13.6%,
# comment volume +61%, cost per review -8.0%. Direction matched, which is the claim.

Offline predicted directory, online confirmed magnitude. That is the claim worth copying.

Running your own reviewer through it

The workflow is self-serve. Sign in with GitHub, register your agent by providing a container image, your configuration and your own model key, then iterate against a 25-PR test set with per-PR detail while you tune. When you are ready, run the full 219 PRs across three rounds and the same judge scores you the way it scores everyone else before you publish to the leaderboard.

Even if you never submit, the structure is worth stealing for internal evaluation. Publish a rubric. Label a dev set with people who did not build the system. Calibrate your grader against those humans and report the agreement number. Version the dataset, judge and matcher. Then track offline movement against online outcomes in the same table, so that when the two disagree you find out before your customers do.