How to Evaluate LLMs Before Production: GitHub's Secret-Scanning Playbook

A language model can perform well on a clean benchmark and still struggle with the cases that matter in production. Benchmarks and curated datasets are useful when prototyping an LLM-based system: they help teams compare models, test an initial prompt, and determine whether an idea is technically plausible. But as a system moves closer to production, the evaluation problem changes. Real inputs are often ambiguous. Labels may be inconsistent. Important context may be missing or truncated. The evaluation set may not reflect the production distribution. GitHub’s team encountered these challenges while evaluating an LLM-based system designed to reduce false positives in GitHub secret scanning, and turned the practices that moved them from promising prototype results to production into a reusable playbook.

1. Define the Decision First, Not the Components

When an LLM system doesn’t perform as expected, the first instinct is often to adjust its technical components: rewrite the prompt, add context, introduce another reasoning step, adjust the surrounding pipeline, or switch models. Before making any of these changes, teams should define the decision the evaluation is meant to support. In the secret-scanning case, the question is: can the system reduce false positives while preserving enough recall to be safe in a production security workflow? To answer it, teams must decide which mistakes are acceptable, which metrics should drive the product decision, and which guardrails must remain within their defined thresholds. In secret scanning, incorrectly suppressing a real credential can be more consequential than asking a developer to review an additional alert. They therefore did not treat precision and recall as equally interchangeable metrics. The primary objective was to reduce false positives and improve precision; recall served as a safety constraint — an experiment could advance only if any decrease remained within a predefined acceptable range.

// Define the decision first. Before changing the prompt,
// adding context, or switching models, decide what the
// evaluation is meant to support. In GitHub's secret
// scanning case: "Can the system reduce false positives
// while preserving enough recall to be safe?"
const DECISION = {
  "question": "reduce false positives while preserving recall",
  "primary": "precision",          // the product goal
  "guardrail": "recall >= 0.95",   // the safety constraint
  "operational": ["latency < 1.5s", "cost per scan <= $0.002"],
};
// Incorrectly suppressing a real credential is worse than
// asking a developer to review an extra alert -- so they
// did NOT treat precision and recall as interchangeable.

2. Change One Variable at a Time, Version Configs Like Code

An LLM-based system continues to change after its first successful evaluation, so evaluation should not be a one-time exercise. Teams revise prompts, adopt new models, change how inputs and context are constructed, and refine the surrounding business logic. Any of these changes can improve the system, introduce a regression, or shift its behavior in an unexpected way. For that reason, they treated offline evaluation similarly to an end-to-end integration test: rerunning it whenever they made a meaningful change to the prompt, model, input construction, or broader system logic. The evaluation also needed to be repeatable enough that each new result could be compared against a known baseline — recording the prompt, model, dataset version, and system configuration for every run. Experiments also need to be designed so the cause of a result is clear: change one major variable at a time and compare each run against a known baseline. They evaluated a prompt revision separately from a model upgrade before testing the two together, because even small prompt changes could shift model behavior, while a model upgrade could affect quality, cost, latency, or output consistency.

// Change one variable at a time, compare against a known
// baseline. Evaluate a prompt revision separately from a
// model upgrade before testing the two together.
const RUNS = [
  { id: "R-001", prompt: "v1", model: "Model A",
    precision: 0.71, recall: 0.78, latency: "1.2s",
    notes: "Baseline" },
  { id: "R-002", prompt: "v2", model: "Model A",
    precision: 0.75, recall: 0.77, latency: "1.2s",
    notes: "Prompt-only change" },
  { id: "R-003", prompt: "v1", model: "Model B",
    precision: 0.74, recall: 0.80, latency: "1.0s",
    notes: "Model-only change" },
];
// If both changed in the same experiment, you would not
// know which one caused the improvement or regression.

3. Keep Offline Evaluation Close to the Production Task

An offline evaluation is only useful when it resembles the task the system will perform in production. In a secret-scanning workflow, the model is rarely evaluating one clean, isolated value. It may need to assess a specific candidate alongside surrounding code and other information that is relevant, incomplete, or potentially distracting. Differences in how that information is presented can materially affect the result. Suppose candidate_value is the value the system is expected to assess. The model may instead focus on example_token because its variable name appears more security-relevant, producing a plausible explanation about the wrong value. This kind of failure is easy to miss when evaluation examples contain only one obvious candidate. The closer the offline pipeline is to the production pipeline, the more useful the evaluation becomes. When the two differ, a strong offline score may simply reflect an easier problem than the one being deployed.

// Treat prompts and evaluation configurations like code:
// version them, record what changed, keep previous configs
// reproducible, make rollback possible.
{
  "eval-config": {
    "version": "2026-08-25.2",
    "prompt": "[email protected]:acme/prompts.git#v14",
    "model": "gpt-5.6-terra@2026-08-20",
    "dataset": "secret-scan-eval@v3",
    "pipeline": "offline-eval@v7",
    "rollback": "git revert <commit>"
  }
}
// Rerun evaluation like an end-to-end integration test
// whenever you make a meaningful change to the prompt,
// model, input construction, or broader system logic.

4. Production Labels Are Signals, Not Unquestionable Truth

Production data can make an evaluation more representative, but its labels often capture workflow outcomes rather than reliable ground truth. A dismissed or resolved secret-scanning alert, for example, does not necessarily represent a false positive. These outcomes may look similar in product data while representing different ground-truth states. Before using production data, ask: does it match the question the evaluation is trying to answer? Are different workflow outcomes being grouped into the same category? For important or ambiguous subsets, you may need to complete a manual review. Synthetic examples, academic benchmarks, and open datasets can help teams bootstrap an evaluation and expand coverage, but these examples should supplement rather than stand in for production-like data. A list of credential strings can test whether a model recognizes common formats, but it cannot fully evaluate how the model reasons about a candidate within real code.

5. Code Walkthrough: Decision, Tracking, Versioning, Proximity, Error Analysis

The code blocks in this post unpack the playbook. Block one defines the decision: primary objective (precision), safety guardrail (recall), and operational constraints (latency, cost) — and why metrics are not interchangeable. Block two is the run-tracking table: R-001 baseline, R-002 prompt-only change, R-003 model-only change, one variable at a time. Block three versions evaluation configs like code: prompt, model, dataset, and pipeline pinned by version with rollback. Block four constructs evaluation examples that preserve the ambiguity and distraction of the real task. Block five is the error taxonomy: group false positives and false negatives by likely source — model, prompt, input, pipeline, dataset, or label — each suggesting a different fix.

// Keep offline evaluation close to the production task.
// In secret scanning the model rarely evaluates one clean,
// isolated value -- it sees a candidate alongside code and
// other context that may be relevant, incomplete, or
// distracting. Preserve that ambiguity.
function buildEvalExample(candidate, surroundingCode) {
  return {
    candidate_value: candidate,   // the value to assess
    context: surroundingCode,     // nearby code (may distract)
    expected: classify(candidate),
  };
}
// Even small differences skew results: a cleaner dataset
// may exclude ambiguous cases, provide more complete
// context, or remove nearby values that distract the
// model. The closer the offline pipeline is to the
// production pipeline, the more useful the evaluation.

6. The Core Lessons

GitHub’s practices distill into five lessons. First, define the decision the evaluation supports before touching components — decide which metric is the goal and which is the guardrail. Second, change one variable at a time and record the prompt, model, and dataset version for every run, so you never attribute an improvement to the wrong change. Third, keep offline evaluation close to the production task — preserve ambiguity, missing context, and distracting information. Fourth, production labels are signals, not truth; manually review important or ambiguous subsets, and use synthetic data to supplement rather than replace. Fifth, aggregate metrics tell you whether a system improved; error analysis tells you what to change next. Manually reviewing dozens or hundreds of examples takes time, but it often leads to faster progress.

// Aggregate metrics tell you whether a system improved;
// error analysis tells you what to change next. Review
// samples of false positives and false negatives and group
// them by likely source.
const ERROR_TAXONOMY = {
  "model": "reasoning about the wrong candidate",
  "prompt": "poor framing or missing instructions",
  "input": "missing context or truncated evidence",
  "pipeline": "wrong data passed to the model",
  "dataset": "labels that do not match the definition",
  "label": "workflow outcome mistaken for ground truth",
};
// A dismissed alert is NOT necessarily a false positive.
// Production labels often capture workflow outcomes rather
// than reliable ground truth -- treat them as signals,
// and for ambiguous subsets, do manual review.