Your Coding Agent Passes 91% of Tests and Ships Secure Code 25% of the Time

On October 10, 2026 a group from the National University of Singapore and Alibaba put a number on something most of us have felt for a year. The Last Secure Code Benchmark (LSCBench) took 150 real vulnerabilities, built 450 tasks out of them, and asked six frontier coding agents to write the fix from a specification that mentions only functionality. Across the board, 73.6% to 91.3% of tasks passed the functional tests. The strongest agent, Claude Fable 5, produced code that was both functional and secure on 25.1% of tasks; the other five landed between 16.4% and 24.7%. On the vulnerability scenarios specifically, the best model cleared 12%. That is not a model review, it is a description of what happens when nothing in the job asks for safety.

A code editor with generated code on screen

1. What LSCBench Actually Measures

The benchmark spans 450 tasks across seven languages — Java, C, C++, JavaScript, Python, PHP and Go — and 24 CWE categories, including 21 of the 2025 CWE Top 25 and 8 of the 2025 Top 10 KEV weaknesses. It cuts each vulnerability three ways: fill in a function (155 generated lines on average), write a whole file (1,782), and build a repository source tree from scratch (4,120). Grading is done by hidden functional and security suites that were validated against both the vulnerable and the fixed version of each scenario. The detail that matters for how you work: every task states functional requirements only and stays neutral about the vulnerability. Nobody tells the agent that the archive importer will be handed a crafted tarball.

# semgrep-rules/agent-defaults.yml
# Three rules that catch the failures agents repeat most often.
rules:
  - id: python-yaml-unsafe-load
    languages: [python]
    severity: ERROR
    message: yaml.load without SafeLoader runs arbitrary constructors.
    patterns:
      - pattern: yaml.load($DATA)
      - pattern-not: yaml.load($DATA, Loader=yaml.SafeLoader)

  - id: python-subprocess-shell-true
    languages: [python]
    severity: ERROR
    message: shell=True turns any concatenated input into command injection.
    pattern: subprocess.$F(..., shell=True, ...)

  - id: python-tar-extract-no-filter
    languages: [python]
    severity: ERROR
    message: tarfile.extractall without filter= allows path traversal.
    patterns:
      - pattern: $T.extractall(...)
      - pattern-not: $T.extractall(..., filter=..., ...)

2. Why the Spec Is the Vulnerability

A coding agent is an extremely literal optimiser. It gets a spec, it writes something that satisfies the spec, and it stops. When the spec says 'load the YAML config from the path the user provides', an implementation that resolves that path anywhere on disk satisfies it perfectly. When the spec says 'unzip the uploaded archive into the working directory', an extract that happily follows `../` satisfies it too. The functional tests pass, because the functional tests encode the same blind spot the spec has. This is why the numbers look contradictory at first glance and are not: the agents are good at the job they were given, and the security work was never in the job. Asking the same model harder questions does not fix a spec that was silent.

## Security contract — append this block to every agent spec

The specification above describes functionality only. These constraints are
part of the contract and are graded separately:

- Treat every byte that arrives from a file, socket, CLI argument or model
  output as hostile. No exceptions for "internal" callers.
- Resolve paths against one declared root. Reject anything that escapes it
  after normalisation, including symlinks and `..` segments.
- Deserialise with safe constructors only. Never instantiate classes named
  in the input.
- Never build shell strings from input. Pass argument arrays.
- Secrets come from the environment. No defaults, no logs, no error text.
- Fail closed: a permission check that errors must deny.

If a requirement conflicts with a constraint, say so instead of choosing.

3. Move the Threat Model Out of the Prompt and Into the Repo

Teams keep trying to fix this with adjectives ('write secure code', 'follow best practices') and it does not survive a long context. What does survive is a file in the repository that says what hostile input means here, plus machine-checkable rules that run whether or not the agent remembered them. Put a security contract next to the spec, and put the same rules in your linter so the violation shows up as a red line instead of a review comment. The rules below are the three that repeat most in agent-written Python: unsafe YAML loading, shell string construction, and archive extraction without a filter.

#!/usr/bin/env bash
# .ci/secure-gate.sh — the whole point is that both halves run before merge.
set -euo pipefail

# 1. Functionality the agent was asked for
pytest -q tests/unit tests/integration

# 2. Security the agent was never asked for
semgrep --config semgrep-rules/ --error --quiet .

# 3. Dependencies the agent quietly added
pip-audit -r requirements.txt --strict

# 4. Secrets the agent inlined "just for now"
gitleaks detect --no-banner --redact

# Anything that only passes step 1 is a demo, not a merge.

4. Run the Functional and Security Gates in the Same Job

The single most common pipeline shape I see is a functional test job that is required for merge, and a security scan that runs nightly on main and is read by nobody. That ordering guarantees the agent's work is judged by the half of the contract it was told about. Move the scan into the merge gate, and make it fail the build, not annotate the diff. On a small service this costs a couple of minutes. The awkward part is not the tooling, it is deciding that a failing static rule blocks a merge — which is exactly the decision that keeps 25% from becoming 16%.

from dataclasses import dataclass
from typing import Callable

@dataclass
class Case:
    name: str              # from a real past incident, not a CVE list
    spec: str              # the request you would actually type
    functional: Callable   # returns True when the feature works
    exploit: Callable      # returns True when the attack still works

def score(agent_fn, cases):
    rows = []
    for c in cases:
        code = agent_fn(c.spec)
        works = bool(c.functional(code))
        unsafe = bool(c.exploit(code))
        rows.append((c.name, works, unsafe))
    n = len(rows)
    return {
        "cases": n,
        "functional_rate": round(sum(w for _, w, _ in rows) / n, 3),
        "safe_rate": round(sum(w and not s for w, s in [(w, s) for _, w, s in rows]) / n, 3),
        "regressions": [r[0] for r in rows if r[1] and r[2]],
    }

# Ten cases from your own incidents beat 450 you cannot reproduce.
# Run it on every harness change; the number moving is the signal.

5. Build a Ten-Case Internal Benchmark and Run It on Harness Changes

You cannot rerun LSCBench against your own code, but you can do something better: turn your ten most recent security-relevant incidents into cases, with a spec that only describes the feature, a functional assertion, and an exploit that should now fail. Then run that suite every time you change the agent, the model, the harness, the prompt template or the tool allowlist. The score you get is not comparable to a published leaderboard and does not need to be. It tells you whether the thing you just changed made your own code worse, which is the only question your release process can act on.

REVIEW = {
    "input_boundary": "Where does untrusted data enter, and what rejects it?",
    "path_handling": "Can any argument, header or filename escape the root?",
    "deserialization": "Which constructors run on attacker-controlled bytes?",
    "shell": "Is there a string-built command anywhere in the diff?",
    "authz": "Which check fails closed when the lookup raises?",
    "secrets": "Did anything land in logs, errors or defaults?",
    "deps": "What was added, and is it pinned and audited?",
}

def gate(diff_answers):
    missing = [k for k, _ in REVIEW.items() if not (diff_answers.get(k) or "").strip()]
    if missing:
        raise SystemExit("review incomplete: " + ", ".join(missing))
    return True

6. What I Would Change This Week

Four things, in order. First, write the security contract and paste it into the spec template the team actually uses, not into a wiki. Second, add the three linter rules and make them block merges. Third, split the CI script so that any agent-produced diff runs functional tests, static rules, dependency audit and secret scan before a human sees it. Fourth, take one incident from last quarter and turn it into a test that an agent would have to pass. None of it requires a new model, and none of it is exciting. That is the point: the benchmark's gap is not closed by better prompting, it is closed by making the unwritten requirement written and checked.