Context Engineering for Coding Agents: What ContextBench Measures and Why Over-Retrieval Fails

ContextBench, submitted on February 11, 2026 by researchers at Nanjing University and University College London, asks a question most agent teams answer by accident: does the agent actually use the context it retrieves? The benchmark spans 1,136 tasks across 66 repositories, and its findings are uncomfortable. State-of-the-art models chase recall and sacrifice precision, so more retrieved context means more noise. Agents frequently inspect the right code and still fail to incorporate it, because retrieval is not utilisation. And balanced retrieval strategies achieve stronger Pass@1 at lower cost. If you build coding agents, this is the most practically useful paper of the year.

Attention is a finite budget

1. What ContextBench Actually Measures

Most coding benchmarks score an outcome: did the tests pass. ContextBench scores the middle of the pipeline. It separates whether the agent retrieved the relevant code from whether it used that code in its final edit, which is exactly the distinction that gets lost when a run fails. The benchmark covers 1,136 tasks across 66 repositories, so the findings are not artifacts of a single codebase or language.

# retrieve.py - over-retrieve wide, then buy precision with a re-ranker
RETRIEVE_K = 40        # cheap recall
FINAL_K    = 8         # expensive precision

def build_context(query, repo_index, reranker):
    candidates = repo_index.search(query, k=RETRIEVE_K)   # recall stage
    ranked = reranker.score(query, candidates)            # precision stage
    return ranked[:FINAL_K]

2. Recall Is Cheap, Precision Is Expensive

The first finding is behavioural: models favor recall over precision. Faced with uncertainty, an agent retrieves more files, more symbols, more history. Each extra chunk feels safe in isolation and is corrosive in aggregate, because the model's attention budget is finite and every irrelevant token competes with a relevant one. More context retrieved means more noise introduced, and that noise is what turns a confident edit into a plausible-looking mistake.

# budget.py - the context window is a budget with a ceiling
TOKEN_BUDGET = {
    "system":    800,
    "task":      600,
    "code":      6000,   # the part that actually fixes the bug
    "examples":  1200,
    "reserve":   800,    # room for the model to reason
}

def fits(section_tokens):
    return sum(section_tokens.values()) <= 9000

def enforce(chunks, cap):
    kept, used = [], 0
    for c in chunks:                 # already sorted by relevance
        if used + c.tokens > cap:
            break
        kept.append(c); used += c.tokens
    return kept

3. Retrieved Is Not Utilised

The second finding is the one teams underestimate. Agents frequently inspect the right code but fail to incorporate it. A file can be read into the context window and still be ignored by the final diff, because the model never connected it to the edit it was making. This is why pass-rate benchmarks can look acceptable while the underlying retrieval is quietly broken: two different failures, retrieval miss and utilisation miss, land in the same bucket.

# filter.py - drop chunks that fail a relevance floor
MIN_RELEVANCE = 0.35

def relevance_filter(chunks, floor=MIN_RELEVANCE):
    return [c for c in chunks if c.score >= floor]

# ContextBench lesson: retrieved is not utilised.
# A chunk that scores below the floor is not context, it is noise,
# and it will compete for attention with the chunk that matters.

Retrieved is not utilised

4. The Context Budget Is a Design Constraint

The practical takeaway is to treat the context window as a budget with a hard ceiling, not as a bucket to fill. A common production pattern follows the benchmark's lesson: over-retrieve at a wide k, then re-rank down to a small final set. Code sample 1 uses k=40 for candidate generation and re-ranks to the top 8, absorbing the cheap recall gain while a re-ranker pays for precision rather than the model.

# rerank.py - a small model that decides what the big model sees
def rerank(query, candidates, cross_encoder, top_k=8):
    pairs = [(query, c.text) for c in candidates]
    scores = cross_encoder.predict(pairs)          # one forward pass per pair
    ordered = sorted(zip(candidates, scores),
                     key=lambda pair: pair[1], reverse=True)
    return [c for c, s in ordered[:top_k]]

5. A Practical Context Pipeline

Assemble it in five explicit stages. Plan what information the task needs. Retrieve wide. Re-rank on relevance to the specific edit. Assemble under a token budget, and verify that what you kept is actually referenced by the change. Code sample 2 keeps the budget honest; code sample 3 drops chunks that fail a relevance floor; code sample 4 shows the re-rank step; code sample 5 turns the whole thing into a measurable eval instead of a vibe.

# eval.py - does the context pipeline actually help?
def context_eval(runs):
    retrieved, used, cost = 0, 0, 0
    for r in runs:
        retrieved += r["chunks_retrieved"]
        used      += r["chunks_in_final_diff"]
        cost      += r["prompt_tokens"]
    return {
        "utilisation_rate": used / max(retrieved, 1),   # the number to track
        "avg_prompt_tokens": cost / max(len(runs), 1),
        "pass_at_1": sum(r["tests_pass"] for r in runs) / max(len(runs), 1),
    }

6. Measure Your Own Context

Do not trust the benchmark, copy its method. Log, for each agent run, how many chunks you retrieved, how many survived re-ranking, how many appear in the final diff, and how many tokens that cost. The ratio of used chunks to retrieved chunks is the number that predicts quality, and it is the number almost nobody tracks. Once you see it, context engineering stops being a prompt trick and becomes ordinary systems work.

7. Why Bigger Windows Do Not Fix It

A million-token context window invites the most expensive mistake in the discipline: dumping the whole repository in and hoping attention sorts it out. It will not. Attention is a finite and roughly zero-sum resource, so every token you add competes with the token that actually matters, and models are demonstrably better at finding relevant text than at ignoring irrelevant text. That asymmetry is why balanced retrieval strategies win in the benchmark even when raw capacity is plentiful. The window is the ceiling, not the strategy. Treat capacity as permission to retrieve well, not as an excuse to retrieve everything.

8. A Pre-Ship Checklist

Five checks before a coding agent ships. One: does every retrieved chunk have a reason to be there, or is it habit? Two: is your retrieval window sized by measurement rather than by the model's maximum? Three: do you log how many chunks survive re-ranking and how many appear in the final diff? Four: does your eval measure utilisation, not just pass rate? Five: when a run fails, can you tell whether it was a retrieval miss or a utilisation miss? Answer those five, and you have a context pipeline instead of a prompt that sometimes works. Skip them, and you have a system whose failures you cannot explain.

Re-rank before you pay for tokens