AI Agent Observability in 2026: Tracing Every Step, Judging Every Trace

Agents fail differently from ordinary software. A traditional service either returns a 200 or throws an error; an agent can return a confident, well-formed, completely wrong answer after making three unnecessary tool calls and one syntactically valid action that did the wrong thing. Binary pass/fail monitoring is blind to all of it. Analysts put instrumentation of GenAI deployments at roughly 15% in early 2026, while Gartner projects that 40% of enterprise applications will feature task-specific AI agents by the end of the year, up from under 5% in 2025. When agents start acting on behalf of your business, why did the agent do that stops being an optional question. The answer requires step-level tracing, continuous evaluation, and span-level cost accounting.

1. Why Agents Need Their Own Observability

A deterministic service is observable with the classic three pillars: metrics, logs, and traces. An agent breaks that model in two ways at once. First, the same input does not always produce the same behavior: temperature, retrieval results, and tool availability all shift the path the agent takes, and the same prompt can trigger a different tool sequence on two consecutive runs -- so you need to observe the distribution of behaviors, not a single happy-path example. Second, failure rarely surfaces as an error: the agent returns something, it is well-formed, it may even be plausible, but it is wrong, or it took an expensive detour, or it called a tool it never needed. None of that trips a 500. That is why step-level tracing is the foundational requirement.

// Span-per-tick: each reasoning step becomes a nested span.
// The four agent operation types from the OTel GenAI spec:
//   create_agent | invoke_agent | invoke_workflow | execute_tool
import { trace } from '@opentelemetry/api';
const tracer = trace.getTracer('agent');
export async function runAgent(task) {
  const workflow = tracer.startSpan('invoke_workflow', { kind: SpanKind.INTERNAL });
  const step = tracer.startSpan('execute_tool', {
    attributes: { 'gen_ai.tool.name': 'search', 'gen_ai.agent.name': 'researcher' }
  });
  const result = await search(task.query);
  step.end(); workflow.end();
  return result;
}
// A five-agent pipeline without this hierarchy is guesswork.

2. Span-per-Tick Tracing: Reasoning as Nested Spans

The core mechanism is span-per-tick tracing: each discrete reasoning step in an agent's execution generates a distinct span inside a distributed trace, nested hierarchically, so a parent trace for a full run contains child spans for every LLM call, tool invocation, memory read, and sub-agent handoff. The OTel GenAI semantic conventions define four agent span operations for this: create_agent, invoke_agent, invoke_workflow, and execute_tool. The subtle part is span kind: invoke_agent is CLIENT when the agent runs remotely (for example OpenAI Assistants API or AWS Bedrock Agent) and INTERNAL when it runs in your own process (LangChain or CrewAI). In multi-agent systems a single INTERNAL invoke_workflow span wraps several invoke_agent children -- that hierarchy is what lets you follow a task across agent handoffs in one trace.

# Two histograms are effectively mandatory: export them or you
# cannot reason about cost or speed.
from opentelemetry import metrics
meter = metrics.get_meter("genai")
latency = meter.create_histogram("gen_ai.client.operation.duration",
    description="Latency of model operations", unit="s")
tokens = meter.create_histogram("gen_ai.client.token.usage",
    description="Token consumption, by input/output")
# For common providers, instrumentation is close to free:
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
OpenAIInstrumentor().instrument()   # semconv spans, no manual code

3. The Standards Layer: GenAI Conventions and MCP Tracing

The most important development in this space is not a product but a specification. The OpenTelemetry GenAI semantic conventions define a shared vocabulary of gen_ai.* attributes across six layers: client model-call spans, agent and workflow spans, MCP conventions, semantic events, metrics, and provider-specific attributes. Two histogram metrics are effectively mandatory for production: gen_ai.client.operation.duration (latency in seconds) and gen_ai.client.token.usage (consumption in tokens, split by input and output). The tool layer used to be the black box; OTel v1.39 added MCP attributes including mcp.method.name, mcp.session.id, and mcp.protocol.version, and when MCP instrumentation detects an outer GenAI span already tracking the tool execution, it enriches that span instead of creating a duplicate -- so an agent calling ten MCP servers traces as cleanly as one calling a single local function.

// The tool layer used to be the black box in agent traces.
// OTel v1.39 added MCP semantic conventions: when MCP
// instrumentation detects an outer GenAI span already tracking
// the tool execution, it ENRICHES that span instead of
// creating a duplicate -- so an agent calling ten MCP servers
// traces as cleanly as one calling a local function.
execute_tool span attributes:
  gen_ai.tool.name: "filesystem"
  mcp.method.name: "tools/call"
  mcp.session.id: "sess_9f2a"
  mcp.protocol.version: "2026-07-28"
// Business metadata on every span (user_id, session_id,
// strategy_id) turns replay into query:
// "tool X failed for users in segment Y over 48h"

4. Continuous Evaluation: The LLM-as-a-Judge Loop

Runtime tracing is the natural complement to offline evaluation frameworks: evaluation catches regressions before deployment, tracing catches what production throws at you afterward. The loop that connects them is LLM-as-a-Judge: running automated judges against sampled production traces to detect semantic drift, factual errors, and policy violations as they emerge, instead of waiting for user complaints. A typical setup judges 5-10% of traces every hour and alerts when the rolling score drops, tool failure rates spike, or a trace drifts from the expected tool sequence. Mlflow and other frameworks now ship this as a structured evaluation layer over live traces.

5. Cost Visibility: Token Spend at Span Level

Token usage tracking at the span level identifies which agent steps consume disproportionate context windows. A single poorly scoped retrieval step can inflate costs by an order of magnitude across millions of runs, and observability makes that visible before it becomes a budget problem. Aggregate gen_ai.client.token.usage by agent and tool, set daily budgets per workflow, and page on overage. Set token budget alerts before you scale: a retrieval agent that works fine at 1,000 runs per day can become expensive fast at 100,000 runs per day if context-window usage is not monitored.

// Continuous evaluation: don't wait for user complaints. Run
// an LLM judge against SAMPLED production traces to catch
// semantic drift, factual errors, and policy violations.
const JUDGE_PROMPT = [
  "You are evaluating an agent trace.",
  "Score 1-5 on: (a) did the final answer satisfy the request,",
  "(b) were all tool calls necessary and correct, (c) is any",
  "claim unsupported by the retrieved context.",
  'Return JSON with "score" and "reason" fields',
].join(NEWLINE); // NEWLINE = the line separator constant
// Schedule: judge 5-10% of traces every hour; alert when the
// rolling score drops, when tool failure rate spikes, or when
// a trace drifts from the expected tool sequence.

6. Build Observability into the Harness from Week One

The most common mistake is treating observability as something you bolt on after launch, and it consistently produces the same outcome: a production incident you cannot explain, a debugging session that takes days instead of hours, and a retrospective where everyone agrees you needed better instrumentation. The shift that works is treating observability as an integrated extension of the agent's control harness: start with span-per-tick tracing on critical agent paths, attach business metadata (user_id, session_id, strategy_id) from day one, and wire up at least one automated evaluation check before production. In-process SDK tracing adds single-digit millisecond overhead per span; a good trace turns a two-day debug into a twenty-minute one.

// Cost guardrails: token usage at span level exposes the
// steps that burn context windows. A single poorly scoped
// retrieval step can inflate cost by an order of magnitude.
{
  "alerts": [
    { "metric": "gen_ai.client.token.usage", "aggregation": "sum",
      "by": ["gen_ai.agent.name", "execute_tool"],
      "condition": "> 2M tokens / 24h", "action": "page-oncall" },
    { "metric": "agent.tool.failure_rate", "condition": "> 0.15",
      "window": "15m", "action": "open-incident" }
  ],
  "budgets": { "retrieval-agent": "10M tokens/day", "coding-agent": "40M tokens/day" }
}
// Export the two mandatory histograms first, then add the
// business metadata. The floor is cost + speed visibility.