83% of Enterprises Must Rebuild Infrastructure for Agentic AI: Inside Google Cloud's 2026 Report

Google Cloud's second annual State of Infrastructure in the Agentic AI Era report opens with a number that should reset your roadmap: 83% of organizations say they require infrastructure upgrades to support production-grade agentic AI. The finding comes from a survey of more than 1,400 senior IT leaders, and it reflects a genuine architectural shift. Enterprise AI has moved from chat, which answers, to agents, which act. Agents that plan, call tools, and execute multi-step tasks put a kind of stress on infrastructure that conversational systems never did. Here is what the report found and how to engineer for it. The report does not argue that agentic AI is overhyped. It argues that the infrastructure underneath it is, and that the gap is measurable.

Legacy infra was built for chat

1. Why Agents Break Chat-Era Infrastructure

The report's core claim is that yesterday's infrastructure was not built for agents that act autonomously. A conversational request is short, stateless, and forgiving. An agentic prompt is the opposite: a single prompt can trigger hundreds of downstream actions, each holding a large context window in memory while it reasons across tools and systems. The report describes production agentic workloads as persistent and stateful, which is exactly what a traditional autoscaling deployment, built to scale to zero between requests, is worst at. Code sample 1 shows the corrected default: treat agent workers as long-lived, stateful services with a warm pool and a shared context store. The uncomfortable part is that this is a capacity-planning problem disguised as an AI problem. If you forecast agent load the way you forecast web traffic, you will size for averages and fail on fan-out.

# agent-worker.yaml — agents are persistent services, not one-shot jobs
apiVersion: apps/v1
kind: Deployment
metadata:
  name: support-agent
spec:
  replicas: 4                       # a warm pool, not cold starts
  template:
    spec:
      containers:
        - name: agent
          resources:
            requests: { cpu: "2", memory: "8Gi" }
            limits:   { cpu: "4", memory: "16Gi" }
          env:
            - name: CONTEXT_STORE
              value: "redis://context-cache:6379"
            - name: SESSION_MODE
              value: "persistent"   # keep long-lived agent state warm
# A single agentic prompt can fan out into hundreds of downstream
# actions, so treat workers as long-lived and stateful.

2. Escape the Inference Tax

Google's research names a specific cost problem: 62% of leaders report a significant inference tax driven by data egress fees, storage bloat, and idle specialized hardware. Meanwhile 81% cite operational complexity as a hidden cost of scaling AI. These are not model costs; they are architecture costs, and they are the most fixable. Code sample 2 keeps the useful context while cutting what you pay to hold it, using prompt caching plus disciplined trimming so a 200,000-token window is a ceiling, not a default. Code sample 3 attacks the tax at its source by co-locating compute and data, since every cross-region byte is a byte you are billed for twice. The inference tax compounds quietly. Egress and idle accelerators are the kind of cost no single engineer owns, which is exactly why it grows until someone charts it.

// context-budget.ts — hold the right context without paying for all of it
const WINDOW = 200_000;              // tokens the model accepts
const HEADROOM = 24_000;             // reserve for tool results + output
type Turn = { role: "user" | "assistant" | "tool"; text: string };

export function trim(history: Turn[]) {
  let budget = WINDOW - HEADROOM;
  const kept: Turn[] = [];
  for (const turn of [...history].reverse()) {
    budget -= Math.ceil(turn.text.length / 4);
    if (budget < 0) break;
    kept.push(turn);
  }
  return kept.reverse();
}
// Google's report calls out massive context windows held in memory.
// Prompt caching plus trimming is how you pay for the useful part only.

3. Match Silicon to the Task

The report's answer to the cost crunch is fluid compute: dynamically matching the right silicon to the right task rather than running everything on one accelerator class. Heavy training benefits from maximum scale, low-latency inference benefits from accelerators purpose-built to maximize on-chip memory so agents can think and react in real time, and general-purpose CPUs are emerging as a critical component for orchestration and control logic. The lesson for platform teams is not to buy more of the biggest GPU. It is to separate training, inference, and orchestration and give each the hardware it actually needs. Fluid compute is really a scheduling discipline. The question is not which chip is fastest, but which chip is right for this specific step in the agent's plan, and whether moving between them costs more than staying put.

# egress-policy.yaml — keep compute and data in the same place
policy: co_locate
rules:
  - store_agent_state: "same_region_as_compute"
  - dataset_reads:
      path: "gs://analytics/*"
      prefer: "same_zone"
  - model_calls:
      cache: "shared_prompt_cache"
      max_retries: 2
cost_guards:
  alert_on_egress_usd_per_day: 250
  block_on_monthly_egress_over: 5000
# 62% of leaders report an inference tax driven by data egress fees,
# storage bloat, and idle specialized hardware. Co-location attacks all three.

4. Scale on Backlog, Not on Average

Agent workloads are bursty in a way that defeats naive autoscaling. Metrics like average CPU utilization look calm while a queue of agent tasks builds up, and a fleet that scales to zero between requests turns every cold start into a lost task. Code sample 4 scales on agent queue depth, keeps a non-zero floor, and slows scale-down with a stabilization window so long-running sessions are not killed mid-task. The report frames this as the need for a resilient, fluid foundation; in practice it means your autoscaler must understand tasks, not just CPU. The stabilization window is the detail teams forget. Aggressive scale-down is fine for stateless request handlers and hostile to an agent holding a twenty-minute session.

# autoscale.yaml — agent fan-out is bursty, not steady
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: support-agent-scaler
spec:
  minReplicaCount: 2            # never drop to zero; cold starts hurt agents
  maxReplicaCount: 40
  triggers:
    - type: prometheus
      metadata:
        query: sum(agent_queue_depth{app="support-agent"})
        threshold: "20"          # ~20 queued agent tasks per replica
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 600   # avoid thrashing long sessions
# One prompt can trigger hundreds of downstream actions; scale on
# backlog, and scale down slowly so in-flight agent sessions finish.

5. Make the Hidden Costs Visible

You cannot optimize an inference tax you never measure. Code sample 5 turns idle accelerator spend and egress into a weekly dashboard, alongside tokens per task, so the two fastest-fix cost leaks are visible to the same team that owns the agents. This is where the report's 81% operational-complexity finding becomes an engineering task rather than a complaint. Once idle spend and egress are on a chart next to agent throughput, the argument for co-location and warm pools stops being architectural taste and becomes arithmetic. Put the dashboard where engineers see it, not where finance does. The team that can co-locate a workload and reclaim egress is the same team reading the chart.

-- inference-tax.sql — turn idle spend and egress into a dashboard
SELECT
  date_trunc('week', ts)                                        AS week,
  SUM(CASE WHEN status='idle' THEN cost_usd END)                AS idle_accel_usd,
  SUM(egress_bytes)/1e9                                         AS egress_gb,
  SUM(egress_cost_usd)                                          AS egress_usd,
  SUM(tokens_in)/NULLIF(SUM(tasks),0)                           AS in_per_task
FROM agent_usage
GROUP BY 1
ORDER BY 1 DESC;
-- 81% of leaders call operational complexity a hidden cost.
-- Make it visible: idle accelerators and egress are the two you can fix
-- fastest with co-location and a warm pool.

6. What to Do First

If the 83% number describes you, the sequence is clear. Move agent workers off scale-to-zero onto stateful warm pools with a shared context store. Put prompt caching and trimming in front of every long-context call. Co-locate compute and data, and alert on egress before your finance team does. Separate training, inference, and orchestration so each runs on the right silicon. Then instrument idle spend and egress so the tax is visible. The report's message is not that agentic AI is too expensive. It is that chat-era infrastructure has a functional ceiling, and the teams that clear it in 2026 will be the ones whose agents actually reach production. The report's headline number is a warning, but also an opportunity: those who fix this first build the platform everyone else runs on.