10 Million Developers and 1,700% More Agent Requests: How to Read Cloudflare's Numbers

On October 20, 2026, Cloudflare published a set of numbers: more than 10 million developers now build on its platform, up from 7.4 million the previous quarter; over the past year, daily requests from AI agents on its network grew more than 1,700%; and this year, for the first time, more than half of Internet traffic was not human. In the same release, VP of product management Rita Kozlov framed the shift this way: the software lifecycle is expanding from developers building for people to developers and agents building for people and agents. Those three numbers are easy to file as headline fodder. Read differently, they describe something concrete: your site may already be serving machines and you would not know it. This post does three things. It explains how each number is measured and where the definitions get soft. It gives you scripts to reproduce the same analysis in your own logs. And it separates what is worth changing from what is not.

Measure your own traffic mix before optimizing for agents

1. The Three Numbers and Their Error Bars

The first is scale: 10 million developers, up from 7.4 million a quarter earlier. That is platform-side accounting, and the denominator is whoever Cloudflare counts as building on it, which they define and you do not. Treat a 35% quarterly jump as a direction, not an industry index. The second is agent traffic: daily requests from AI agents up more than 1,700% in a year. There is an important qualifier buried in it, namely how an AI agent request is identified, whether by user agent, behavioral signature or verified identity. Different methods produce very different numbers. The third is the easiest to misread: this year, for the first time, more than half of Internet traffic was not human. Note the subject, which is Cloudflare's network, and note that non-human includes crawlers, uptime probes, API clients, compromised IoT devices and agents. Folding all of that into a new agent era is over-reading. The useful move is to take the three numbers as evidence that demand-side change is real, then go measure your own logs.

# 1) Classify your own traffic before you believe anyone's percentage.
import re
from collections import Counter

NAMED_AGENT = re.compile(
    r"(GPTBot|OAI-SearchBot|ClaudeBot|Claude-User|PerplexityBot|"
    r"Google-Extended|Amazonbot|Applebot-Extended|meta-externalagent|"
    r"Bytespider|CCBot)", re.I)
BROWSER = re.compile(r"(Mozilla|Chrome|Safari|Firefox|Edg)", re.I)

def classify(ua: str) -> str:
    if not ua:
        return "unknown"
    if NAMED_AGENT.search(ua):
        return "named-agent"
    if BROWSER.search(ua):
        return "browser"
    if re.search(r"(python-requests|httpx|aiohttp|curl|go-http-client|okhttp)", ua, re.I):
        return "script"
    return "other"

# Feed it your access log lines and look at the distribution.
print(Counter(classify(ua) for ua in user_agents).most_common())

2. Why Half Non-Human Traffic Matters Both Ways

The worrying half is concrete. If you designed pages for humans with large images, motion, waterfall dependencies and three round trips before first paint, an agent that asks once a second multiplies your bandwidth and upstream compute while never clicking an ad, signing up, or reading a recommendation slot. The exploitable half is equally concrete. An agent comes to your site because it needs a specific fact: which parameter a tool takes, what a format requires, what a field means. The answer is often a few hundred bytes while the page that displays it weighs four hundred kilobytes. Human readers need the packaging. Machines need the data. Serving both from one endpoint is where the cost problem starts, and the real question is not whether to welcome agents but which part of your content is factual and can be handed over cheaply.

# 2) Aggregate hides cost. Break share down by agent AND by path.
from collections import defaultdict

stats = defaultdict(lambda: {"hits": 0, "bytes": 0})

for row in access_log:                       # ts, ua, path, bytes_sent
    kind = classify(row["ua"])
    if kind != "named-agent":
        continue
    agent = NAMED_AGENT.search(row["ua"]).group(1).lower()
    stats[(agent, row["path"])]["hits"] += 1
    stats[(agent, row["path"])]["bytes"] += row["bytes_sent"]

for (agent, path), s in sorted(stats.items(), key=lambda kv: -kv[1]["bytes"])[:15]:
    mb = s["bytes"] / 1e6
    print(agent.ljust(22), path.ljust(32), str(s["hits"]).rjust(8), round(mb, 1), "MB")

# The pattern that matters: are agents pulling your heaviest HTML pages
# thousands of times, or asking for a small number of specific facts?

3. Measure First: Classify Your Logs

The first snippet classifies user agents into named agents such as GPTBot, ClaudeBot and PerplexityBot, browsers, script clients like python-requests and curl, and everything else. The second keeps only named agents and aggregates by agent name and path, reporting hit counts and transferred bytes, sorted by bytes. The point is to turn a vague percentage into three conclusions you own: who comes, how often, and how much they take. In practice you tend to see three shapes. Some agents fetch only the homepage and sitemap, which is indexing. Some repeatedly fetch the same set of tool pages, which is answer retrieval. A third group hits only API endpoints, which means someone wired your site into a pipeline. The response differs completely across the three, and you cannot optimize before you separate them.

// 3) Give agents a cheap, machine-readable surface instead of a 40KB page.
//    One JSON endpoint can replace a thousand full HTML renders.
export default {
  async fetch(req: Request): Promise<Response> {
    const url = new URL(req.url);
    const slug = url.searchParams.get("tool");
    if (!slug) {
      return Response.json({ error: "tool parameter required" }, { status: 400 });
    }
    const meta = await getToolMeta(slug);       // your existing lookup
    if (!meta) {
      return Response.json({ error: "not found" }, { status: 404 });
    }
    return Response.json(
      { slug: meta.slug, title: meta.title, category: meta.category, updated: meta.updatedAt },
      { headers: { "Cache-Control": "public, max-age=3600" } }
    );
  },
};

// Same facts, a fraction of the bytes, and cacheable at the edge.
// Document it in llms.txt so well-behaved agents find it instead of scraping.

4. Give Agents a Cheap Lane, Not a Gate

The third snippet is a tiny JSON endpoint: one tool slug in, title, category and updated timestamp out, with Cache-Control so the edge can serve it. It does not replace the page; it gives callers who only need facts a cheap path. Two supporting moves: document it in llms.txt or your docs so well-behaved agents use the front door instead of scraping HTML, and include a last-updated timestamp so callers can judge whether to refetch. The second snippet already told you which paths get hammered; those are the ones that deserve a machine-readable version. One counterintuitive detail: this endpoint saves your money, not the caller's. The agent's bill is paid by whoever runs it, while your bandwidth and upstream compute are paid by you, which makes this a pure cost reduction that requires no cooperation.

# 4) Quotas belong to identities, not IPs. Agent fleets rotate addresses.
import time
from collections import OrderedDict

class Window:
    def __init__(self, limit: int, seconds: int = 60):
        self.limit, self.seconds = limit, seconds
        self.seen = OrderedDict()

    def allow(self, identity: str) -> bool:
        now = time.time()
        hits = [t for t in self.seen.get(identity, []) if now - t < self.seconds]
        hits.append(now)
        self.seen[identity] = hits
        self.seen.move_to_end(identity)
        return len(hits) <= self.limit

browser = Window(limit=120)      # a human browsing a few tool pages
agent   = Window(limit=30)       # a crawler that respects the contract

# Identity = verified agent name (or API key), never the source IP.
# Return 429 with Retry-After so well-behaved agents back off instead of retrying hard.

5. Quotas by Identity, Not by IP

The fourth snippet is a sliding-window counter keyed by identity. Why not IP? Because legitimate agents run in clouds with rotating egress pools, so an IP you block today belongs to an unrelated machine tomorrow, and determined scrapers learned to rotate long ago. Identity means a signed agent identity first if the caller offers one, then an API key, and only then user agent plus rate limiting. The limiting itself should be polite: return 429 with Retry-After so a well-behaved agent backs off instead of retrying harder. Human visitors deserve a visibly higher threshold, because a person clicking through a dozen tool pages in a minute is normal. The code is under twenty lines, and it turns the question of whether agents show up from an emotional one into a configuration one.

# 5) Price agent traffic as its own line item. Blended averages lie.
report = {
    "human":   {"requests": 0, "egress_gb": 0.0, "compute_ms": 0},
    "browser": {"requests": 0, "egress_gb": 0.0, "compute_ms": 0},
    "agent":   {"requests": 0, "egress_gb": 0.0, "compute_ms": 0},
}

for row in access_log:
    bucket = report[classify(row["ua"])] if classify(row["ua"]) in report else report["human"]
    bucket["requests"] += 1
    bucket["egress_gb"] += row["bytes_sent"] / 1e9
    bucket["compute_ms"] += row.get("upstream_ms", 0)

for name, r in report.items():
    avg = r["egress_gb"] / r["requests"] if r["requests"] else 0
    print(name.ljust(8), str(r["requests"]).rjust(9), round(r["egress_gb"], 2), "GB",
          round(avg * 1000, 1), "KB/req")

# If agent requests are cheap per call and high in volume, the fix is caching
# and a JSON surface. If they are expensive per call, the fix is the endpoint.

6. Price Agent Traffic Separately, Then Talk About the Edge

The fifth snippet splits traffic into human, browser and agent buckets, counting requests, egress gigabytes and upstream compute milliseconds, then printing average bytes per request. This looks like bookkeeping and is actually an architecture decision. If agent requests are cheap per call but high in volume, the fix is caching and a JSON surface. If they are expensive per call, the agent is triggering heavy computation, and the fix is moving that computation into an offline artifact. This is where Kozlov's emphasis on Workers running on isolates rather than virtual machines or containers becomes a cost statement: high-frequency, short, stateless agent requests are cheapest on a per-request model with near-zero cold start, while an architecture that spins a container per request will simply hand you a larger bill on agent traffic. Optimizing for agents usually means moving the expensive step out of the request path, not buying more machines.