Your AI Agent Is Only as Good as Its Harness: Tool Contracts, Permissions & Error Taxonomies

The New Stack published a refreshingly practical piece on August 30, 2026: "Your AI agent is only as good as the harness around it." The opening is a blunt truth: most agent projects are much harder than the demos suggest. In the demo, everything goes right — the user asks a question close to the one from the demo, tools return success, policies did not change. Then a real user shows up: the question is worded slightly differently, the account record is incomplete, a tool returns an error, a policy changed last week, or the agent discovers a capability boundary — it can read an invoice but cannot change it. This is often where the real work begins. The core claim: "The model is one part of the service. The agent harness is the rest — the scaffolding the application builds around the model to feed it the right inputs and check its outputs, helping catch failures before they spread."

1. Why Demo Magic Does Not Survive Production

Developers already know the idea of a test harness, which wraps code to run under controlled conditions. A production agent needs the same wrapper. Good model output matters, but it does not prove an agent is ready for real work. Proving that is the harness's job: tool contracts that limit what a wrong call can do, permissions enforced outside the model even when an instruction attempts to bypass them, context paths and trace records the team can actually inspect, and tests built from the failures users will find first. As the article says: "Get those right, and the demo magic starts surviving contact with production."

// A tool contract from The New Stack's billing example:
// input/output schemas, a timeout, and DEFINED error
// states. The idempotency key and the error split do a
// lot of work in that schema.
{
  "name": "update_billing_plan",
  "description": "Apply a previously quoted plan change to an account.",
  "input": {
    "account_id": "uuid (server-verified)",
    "quote_id": "uuid",
    "idempotency_key": "uuid"
  },
  "output": {
    "status": "applied | rejected",
    "effective_date": "date"
  },
  "timeout_ms": 5000,
  "errors": {
    "retryable": ["RATE_LIMITED", "UPSTREAM_TIMEOUT"],
    "terminal": ["QUOTE_EXPIRED", "APPROVAL_REQUIRED", "ACCOUNT_NOT_FOUND"]
  }
}
// "An error message is a prompt." -- The New Stack

2. Tool Contracts: Define the Boundaries of Every Tool

Tools are where the harness meets your production systems, so they come with contracts. An agent tool is an API for a caller that can make incorrect choices: a short tool description helps the model choose the right tool, but it does not protect the API from invalid input or unsafe requests. Give each tool a specific job with input and output schemas, a timeout, and defined error states. The billing tool example from the article (block one) is the template: input declares account_id, quote_id, and an idempotency key; output declares status and effective date; timeout is 5000ms; errors are explicitly split into retryable (RATE_LIMITED, UPSTREAM_TIMEOUT) and terminal (QUOTE_EXPIRED, APPROVAL_REQUIRED, ACCOUNT_NOT_FOUND).

// Contract validation middleware: the harness checks
// every tool call BEFORE it reaches production systems.
// The model can make incorrect choices; the contract
// cannot be bypassed by a clever instruction.
function enforceContract(contract, args) {
  for (const [field, type] of Object.entries(contract.input)) {
    if (args[field] === undefined) {
      throw { code: "MISSING_FIELD", field, hint: "Provide " + field };
    }
    if (type.includes("uuid") && !isUuid(args[field])) {
      throw { code: "INVALID_UUID", field };
    }
  }
  if (!args.idempotency_key) args.idempotency_key = crypto.randomUUID();
  return args;
}
// Permissions live OUTSIDE the model. Even if an injected
// instruction says "skip the check", this code runs before
// the tool call and cannot be prompted away.

3. An Error Message Is a Prompt

The most quotable line in the article: "An error message is a prompt." The model reads whatever your tool returns and acts on it. ERR_422 teaches the agent nothing. APPROVAL_REQUIRED: annual plan changes need human sign-off tells it exactly what to do next. So the error split is not a detail: retryable versus terminal decides whether the agent loops forever or hands off cleanly. Block three demonstrates the error taxonomy handler: retryable errors retry with backoff (the idempotency key guarantees no duplicate execution), approval-required errors enter a human approval queue, and terminal errors return with guidance.

// Error taxonomy handler: an error message is a prompt.
// "ERR_422 teaches the agent nothing. APPROVAL_REQUIRED:
// annual plan changes need human sign-off tells it
// exactly what to do next."
async function handleToolError(err, ctx) {
  const isRetryable = err.retryable?.includes(err.code);
  if (isRetryable && ctx.attempts < 2) {
    await sleep(backoff(ctx.attempts)); // 500ms, 2s
    return retry(ctx);                   // idempotent: same key
  }
  if (err.code === "APPROVAL_REQUIRED") {
    await enqueueHumanApproval({ task: ctx.task, reason: err.message });
    return { status: "awaiting_approval", ticket: ctx.ticket };
  }
  return { status: "failed", code: err.code, guidance: err.hint };
}
// Retryable vs terminal is not a detail: it decides
// whether the agent loops forever or hands off.

4. Permissions Enforced Outside the Model

The model's instructions can come from anywhere — including injected hostile content. So permissions cannot live in the prompt; they live in code outside the model: even when an instruction attempts to bypass them, the check runs before the tool call and cannot be prompted away. Block four shows an action-resource policy: read, write, approve, and delete each have allowed scopes, and payment:apply requires the finance-approver role. Every authorization writes an audit log — agent identity, action, resource, timestamp. The article's phrasing: "permissions enforced outside the model even when an instruction attempts to bypass them."

5. Trace Records the Team Can Actually Inspect

The harness's third job is making agent behavior inspectable: context paths (what data the agent saw, in what order) and trace records (which tools it called, with what results). Without these, when something breaks you are staring at a black box that says "it did the thing." The article also notes: if you are defining tools through MCP, some schema plumbing may be handled for you — but the contract itself is still yours to define, including timeouts, error taxonomy, and idempotency behavior. Tool contracts, permissions, and traces together answer "is this agent ready for production."

// Permissions enforced outside the model: even when an
// instruction attempts to bypass them, the harness checks
// the ACTION, not the model's intent.
const ACTION_POLICY = {
  "read": ["account", "docs", "repo"],
  "write": ["draft", "ticket"],
  "approve": [],        // never, without human
  "delete": [],         // never, without human
  "payment:apply": ["finance-approver"],
};

function authorize(agentId, action, resource) {
  const allowed = ACTION_POLICY[action] || [];
  if (!allowed.includes(resource)) {
    throw { code: "FORBIDDEN", hint: resource + " requires a different role" };
  }
  auditLog({ agentId, action, resource, at: Date.now() });
}
// "Permissions enforced outside the model even when an
// instruction attempts to bypass them." -- The New Stack

6. Tests Built from the Failures Users Will Find First

Finally, the harness needs tests — built from the failures users will find first, not synthetic happy paths. Block five shows regression tests for the billing agent: an expired quote must not be applied, retries must reuse the same idempotency key, and the payment API must not be called when approval is missing. A benchmark score can measure response quality, but these tests prove the agent cannot break your systems. The closing summary is worth remembering: "The model supplies the reasoning, and the harness supplies the boundaries the model doesn't have on its own." In 2026, getting the harness right is the core of agent engineering.

// Tests built from the failures users will find first:
// the harness gets a regression suite that replays real
// failures, not synthetic happy paths.
describe("billing agent harness", () => {
  it("does not apply a quote that expired", async () => {
    const res = await runAgent("apply quote Q-99", {
      toolState: { quote: { id: "Q-99", status: "EXPIRED" } },
    });
    expect(res.status).toBe("failed");
    expect(res.code).toBe("QUOTE_EXPIRED");
    expect(finance.apply).not.toHaveBeenCalled();
  });

  it("retries with the SAME idempotency key", async () => {
    const key = "op-1";
    await runAgent("update billing", { idempotencyKey: key });
    expect(finance.apply).toHaveBeenCalledWith(expect.objectContaining({ idempotencyKey: key }));
  });
});
// "Tests built from the failures users will find first."
// Benchmark scores measure quality; these tests prove the
// agent cannot break your systems.