57% of Enterprises Now Spend More on Telemetry Than on the Infrastructure It Watches (and 56% Cannot Measure Answer Quality)

On October 20, 2026, Cisco published the sixth edition of Splunk's State of Observability research, built on 2,050 ITOps and engineering respondents across 10 countries. The headline finding is not subtle: the urgency to deploy AI agents has outrun the ability to monitor them. Jeetu Patel, Cisco's President and Chief Product Officer, puts the operational version of it plainly — most companies will soon have far more agents at work than people, and managing agents at scale means reimagining both security and observability, because in the agentic era "have our agents been compromised" and "are they doing their job well" are inseparable questions moving at machine speed. What the report adds is quantification for that inseparability.

57% say the observability bill now exceeds the cost of what is being observed.

1. High priority, no business outcome attached

87% of respondents confirm that monitoring AI is a priority for their organization, a number so high it barely needs discussing. The next two lines are where the problem lives: 54% do not have clear business outcomes tied to it, and 56% say they are unable to measure AI workload metrics such as LLM response quality. In other words most teams are already spending money to monitor AI without having defined what successful monitoring looks like. The report's explanation is structural. Monitoring traditional applications means watching latency, error rates and saturation. Monitoring an agent requires a different class of behavioural metrics tied to business outcomes and the agent's decisions — and that class is the hardest to define, which makes it the easiest to skip.

# One span per agent run, with attributes you can actually bill against.
# Names follow the OpenTelemetry GenAI semantic conventions where they exist.

span_name: invoke_agent
attributes:
  gen_ai.operation.name: invoke_agent
  gen_ai.agent.id: reviewer-code-quality      # which agent
  gen_ai.agent.version: 2026-10-05             # which build
  gen_ai.request.model: claude-sonnet-5.5
  gen_ai.request.effort: high                  # effort IS a cost input
  gen_ai.usage.input_tokens: 184320
  gen_ai.usage.output_tokens: 6120
  gen_ai.usage.cache_read.input_tokens: 1041880
  gen_ai.usage.cache_creation.input_tokens: 72400
  app.workload: code-review                    # the billable unit
  app.tenant: acme
  app.outcome: posted_review                   # success | timeout | denied
  app.tool_calls: 16
  app.retries: 1

# Per-run attributes are what turn a telemetry lake into a cost model.
# Session-level aggregates only tell you that you spent money.

2. Hallucination visibility: 85% call it important, 58% can do it

One pairing from the report belongs on the wall above the monitoring dashboard. 85% say it is important to have visibility into AI hallucinations; only 58% claim to be effective at it. That gap is not an attitude problem, it is a tooling problem, because no P99 latency chart will tell you that a fluent answer was invented. It also explains why response quality is a blank in 56% of organizations. Measuring it requires sampling, a judge, and storing the score on the same run as the cost — three things classic APM does not do.

-- Attribute spend to an agent, a model and a workload, from telemetry
-- you already store. Cache writes and reads must be priced separately
-- or the number is wrong by a factor of ten.
WITH runs AS (
  SELECT
    span_attributes['app.workload']                   AS workload,
    span_attributes['gen_ai.agent.id']                AS agent_id,
    span_attributes['gen_ai.request.model']           AS model,
    sum(CAST(span_attributes['gen_ai.usage.input_tokens'] AS BIGINT))            AS in_tok,
    sum(CAST(span_attributes['gen_ai.usage.output_tokens'] AS BIGINT))           AS out_tok,
    sum(CAST(span_attributes['gen_ai.usage.cache_read.input_tokens'] AS BIGINT)) AS cache_read,
    sum(CAST(span_attributes['gen_ai.usage.cache_creation.input_tokens'] AS BIGINT)) AS cache_write,
    count(*) AS runs
  FROM otel_traces
  WHERE date >= current_date - INTERVAL '7' DAY
    AND span_attributes['gen_ai.operation.name'] = 'invoke_agent'
  GROUP BY 1, 2, 3
)
SELECT
  workload, agent_id, model, runs,
  round(
      in_tok      / 1e6 * price_input
    + out_tok     / 1e6 * price_output
    + cache_read  / 1e6 * price_cache_read
    + cache_write / 1e6 * price_cache_write
  , 2) AS usd_7d,
  round(
      (in_tok      / 1e6 * price_input
     + out_tok     / 1e6 * price_output
     + cache_read  / 1e6 * price_cache_read
     + cache_write / 1e6 * price_cache_write) / runs
  , 3) AS usd_per_run
FROM runs JOIN price_pivot USING (model)
ORDER BY usd_7d DESC;

-- Which of these numbers moved week over week is the actual alert.

3. Cost: the bill is not too big, it is unpredictable

The cost section is the most informative part of the report. 55% say they cannot control AI costs, because consumption is rising while usage stays hard to predict. 60% say unexpected billing spikes cause more operational strain than high baseline costs — note the shift in wording: the pain is uncertainty, not price. And 57% say their telemetry costs have eclipsed the cost of the infrastructure being monitored. That sentence is worth pausing on: observability exists to make systems governable, and it has become a line item that needs governing itself. The report's suggested direction is a unified view, and 67% of respondents use a single observability platform precisely so that they can pinpoint where telemetry costs originate.

One agent session produces traces, logs, tokens, tool calls and dollars. They rarely land in the same place.

# Quality is a metric, not a vibe. Sample outcomes, score them,
# and store the score next to the cost of producing it.

def record_eval(run_id, agent_id, sampled_output, reference):
    score = judge(sampled_output, reference)   # 0..1, rubric-based
    emit_metric(
        name="agent.quality.score",
        value=score,
        attributes={
            "agent.id": agent_id,
            "run.id": run_id,
            "eval.model": judge.model,     # judges cost money too
            "sampled": True,
        },
    )

# Sampling policy that keeps the signal and cuts the volume:
# - 100% of failures, denials and timeouts
# - 100% of runs above a cost threshold
# - 2-5% of the remainder, stratified by agent and workload
# Then chart quality against cost per run on the same axis.
# A cost drop with a flat quality line is a win; a cost drop with a
# falling line is a regression you would otherwise call an optimisation.

4. Security and observability are converging

Once agents hold access across the whole stack, one wrong action becomes a business failure, and the historic wall between observability and security starts to come down. 83% of respondents are concerned about the unintended consequences of AI agents. 81% say monitoring AI workloads has increased collaboration between observability and security teams. 73% can accurately trace performance issues back to security root causes, up from 65% a year earlier. 77% of observability teams now troubleshoot together with security teams, up from 62% in 2025, and data sharing between the two has climbed from 74% to 84%. Read together, the numbers say something simple: in the agentic era these are two views of one problem, and separating them loses on both sides.

# Telemetry that costs more than the system it watches is a design bug.
# Split by signal type and drop what nobody queries. Nothing below is a
# vendor default; it is a policy that has to be written down.
retention:
  traces:
    agent_runs: 30d            # expensive, sampled, keep short
    tool_calls: 7d             # high cardinality, low per-row value
  metrics:
    agent.quality.score: 400d  # cheap, keep long
    agent.cost.usd_per_run: 400d
  logs:
    agent_transcripts: 14d     # redact secrets before storage
sampling:
  tail:
    keep_if:                   # evaluate once the trace is complete
      - status == error
      - cost_usd > 0.50
      - attribute.app.workload in ('billing', 'security-review')
    default_rate: 0.03
cardinality:
  forbid_labels:
    - prompt_text       # never a label: unbounded values
    - user_email
    - file_path

5. What the leader cohort does, and the four instrumentation decisions that follow

The research carves out a group of leaders — organizations achieving concrete business outcomes with observability — and describes three practices: they rely on AI agents to fix and prevent issues frequently, they closely monitor the factors that keep AI in check (model quality, cost, usage, security, privacy), and they connect signals to business impact. The outcomes diverge sharply. 91% of leaders can accurately attribute AI costs, against 69% of non-leaders. Leaders are 2.4 times more likely to say the ROI of their observability platform significantly exceeds expectations, and 54% say observability has yielded at least $5 million in annual business revenue, versus 28% of non-leaders. Translated into engineering decisions, four things matter. First, instrument per run: one span per agent invocation carrying agent id, version, model, effort, input and output tokens, cache read and write tokens, tool call count, retries and outcome — without those attributes a telemetry lake never becomes a cost model. Second, price cache reads and cache writes separately, or the number will be wrong by an order of magnitude. Third, make quality a metric, and treat the sampling policy as more important than the judge: keep every failure, denial, timeout and over-threshold run, then stratify 2% to 5% of the rest by agent and workload. Fourth, govern the telemetry itself — per-signal retention, tail sampling, and a hard ban on prompt text and user identifiers as labels. None of it is free, but it is considerably cheaper than learning about a spike from the invoice.

84% now share observability data with security teams, up from 74%.

# The alert set that matches what the survey says is missing.
alerts:
  - name: agent_cost_spike
    expr: >
      sum by (agent_id) (rate(agent_cost_usd_total[1h]))
      > 1.5 * avg_over_time(
          sum by (agent_id) (rate(agent_cost_usd_total[1h]))[7d:1h] offset 1h
        )
    for: 30m
    why: 60% say surprise spikes hurt more than a high baseline

  - name: quality_regression
    expr: avg_over_time(agent_quality_score[24h]) < 0.75
    for: 2h
    why: 56% cannot measure LLM response quality today

  - name: unattributed_spend
    expr: >
      sum(agent_cost_usd_total) - sum(agent_cost_usd_total{agent_id!=""})
      > 20
    why: cost with no agent, model or workload attached is precisely
         the number leaders can attribute and non-leaders cannot