New Relic AI Evaluation: Judging Agents by Business Outcome, Not Just LLM Calls

New Relic introduced AI Evaluation on October 6, 2026, a new capability inside its AI Observability platform. The design bet is blunt: evaluating an AI application by looking at isolated single-LLM calls is not enough. AI Evaluation works at the transaction level, linking response quality and model behaviour to distributed traces, so an engineer can tell whether a failure began in the prompt, the retrieval layer, the model, or the backend - and see what it cost in business terms. It is a sign that observability vendors now assume agent traffic is the norm.

AI Evaluation works at the transaction level, not the LLM call

The problem with evaluating single LLM calls

Most AI evaluation today is a point solution: you send a prompt, you grade the response, you move on. It is useful, and it is insufficient. A production AI application is not one call - it is a chain of them, wrapped in retrieval, tool use, and backend services, all chained to a user's goal. When that goal fails, grading a single call tells you a number without telling you a cause. New Relic AI Evaluation starts from the opposite assumption: if you want to know why an AI feature failed, you have to look at the transaction, not the call.

# The core idea: evaluate at the transaction, not the LLM call.
# A transaction is the whole user goal, end to end.

EVALUATION = {
    'unit': 'transaction',        # not a single prompt/response
    'links': ['response_quality', 'model_behaviour', 'traces'],
    'measures': ['business_impact', 'guardrail_performance'],
    'scope': 'developer-to-production lifecycle',
}

What transaction-level evaluation means

AI Evaluation assesses AI behaviour across the full developer-to-production lifecycle, at the level of the transaction - the end-to-end unit a user actually cares about. It links response quality and model behaviour to distributed traces, which is what turns a vague quality score into a diagnosis. Instead of knowing that an answer was wrong, you can see whether the prompt lacked context, the retrieval layer surfaced the wrong documents, the model hallucinated, or a backend service returned an error. That distinction is the difference between a dashboard and a debugging tool.

# Emit OpenTelemetry spans for every step, tagged with a user ID
# so an agent action can be tied to a downstream outcome.

span = tracer.start_span('agent.turn')
span.set_attribute('user.id', user_id)
span.set_attribute('agent.step', step_name)
span.set_attribute('retrieval.docs', len(docs))
span.set_attribute('model.name', model)
span.set_attribute('tokens.in', usage.prompt_tokens)
span.set_attribute('tokens.out', usage.completion_tokens)
span.end()
# One trace shows which layer actually caused the failure.

Distributed traces reveal which layer caused a failure

Guardrail performance in real time

A quieter but important piece: the feature provides automated, real-time insights into AI guardrail performance. Guardrails - the filters and policies that constrain what a model may do - are usually treated as a set-once configuration. In practice they need monitoring like any other production control. If a guardrail starts tripping far more often than usual, that is a signal: either the traffic changed, or the guardrail is misfiring, or the model behind it shifted. Surfacing that rate live is how teams keep trust in the very controls meant to protect them.

# Attribute a failed transaction to the layer that caused it.

def diagnose(trace, expected):
    if trace.prompt_missing_context(expected):
        return 'prompt'
    if trace.retrieval_missed(expected):
        return 'retrieval'
    if trace.model_refused_or_hallucinated(expected):
        return 'model'
    if trace.backend_error():
        return 'backend'
    return 'unknown'

Part of a bigger release

AI Evaluation was not a lone announcement. New Relic's broader release also included the Ground Truth command-line interface and Infrastructure 360, both aimed at putting observability data more directly into developers' and operators' hands. Read together, the direction is clear: the vendor is betting that AI workloads are now the main event, not a sidecar, and that the tooling around them should sit inside the same observability platform as everything else rather than in a separate, AI-only silo.

# Guardrail performance is a first-class metric, monitored live.

def guardrail_alert(traffic):
    trips = [t for t in traffic if t.guardrail_tripped]
    rate = len(trips) / max(len(traffic), 1)
    if rate > 0.02:            # more than 2% tripping
        page_on_call('guardrail-trip-rate', rate)
    return rate

# Guardrails that silently fire are guardrails nobody trusts.

Guardrail performance is monitored as a first-class metric

Why this matters for agent teams

If you run agents, the practical takeaway is that outcome and trace are two halves of one measurement. A success rate without traces cannot tell you what to fix; traces without an outcome metric cannot tell you whether the fix mattered. AI Evaluation is notable less for any single feature than for codifying that pairing: judge the transaction for whether it worked, and read the trace to learn why. Teams that adopt that habit early will find the debugging loop shortens dramatically, because they stop guessing at the layer and start seeing it.

# Wire evaluation into CI so regressions never reach production.

def ci_gate(eval_run, threshold=0.9):
    if eval_run.quality_score < threshold:
        raise SystemExit('Blocked: quality regressed below gate')
    if eval_run.guardrail_trip_rate > 0.02:
        raise SystemExit('Blocked: guardrails tripping too often')
    return 'ship'

A practical observability checklist

Five things you can do whether or not you adopt this product. One: instrument every agent step with OpenTelemetry spans and a user ID, so outcomes can be attributed. Two: define a small set of transaction-level success metrics tied to real goals. Three: monitor guardrail trip rates as a live metric with an alert threshold. Four: capture token and latency cost per transaction, not per call. Five: gate deploys in CI on quality and guardrail regressions. None of this requires a specific vendor - but all of it requires treating AI features as systems, not as prompts.