Same Game, New Rules - How Agentic Traffic Rewrote Bot Defense

For decades, website operators have been fighting the same battles.

• Scraping - Competitors and aggregators systematically harvest product data, pricing, and content at a scale no human could match, eroding competitive advantage and inflating infrastructure costs.
• DDoS - Volumetric floods overwhelm origin servers and impact businesses of every size, from enterprise SaaS to indie storefronts.
• Credential stuffing - Automated campaigns that test billions of username/password pairs leaked from third-party breaches and compromise accounts at machine speed.
• Ad fraud and inventory abuse - Synthetic traffic that burns ad budgets and distorts analytics, or bots that snatch limited inventory before actual users get a chance.

‍

To meet these challenges, the security industry responded with CAPTCHAs, behavioral fingerprinting, rate limiting, device reputation signals, and bot management platforms. All these solutions emerged as ways to separate human traffic from automated abuse. And all were based on a simple underlying thesis: that real users behave differently from bots, and that the gap between human and bot could be detected and enforced.

That logic no longer holds. The mass adoption of AI agents - tools that browse, transact, and interact on behalf of users - has complicated the human-vs-bot distinction. But the key is this: the attacks have not changed; the bots did.

‍

In this post, we’ll look at why security challenges haven’t really changed in the era of AI, why our defenses can't see the threats, and how we need to adapt to rectify this.

‍

The Old Problem: Thirty Years of Human vs. Bot

‍

For three decades, bots were limited, imperfect, and left fingerprints at every layer of the stack. Each defense that emerged targeted a different one:

‍

• Signature and reputation rules (the WAF). The earliest and bluntest method: match known-bad User-Agents (curl, python-requests, Googlebot imposters), known-bad IP ranges, and known-bad request patterns. WAFs are cheap and fast, but also easy to evade by changing a string. Despite this, they still catch a surprising amount of low-effort traffic.
• Rate limiting and volumetric controls. If one IP asks for ten thousand pages an hour, you don't need to know what’s driving it. You just need to block it. This method is effective against brute force and scraping floods - at least until attackers spread the load across thousands of IPs (botnets, proxy pools) so no single source looks abnormal.
• Challenge-response (CAPTCHA / Turing tests). This is the philosophical heart of  human-bot differentiation: pose a task presumed easy for humans and hard for machines.  Each generation - distorted text, image grids, invisible risk scoring - got solved and replaced by a harder one, at rising cost to real users.
• Browser fingerprinting. Rather than challenge the visitor, this method measures them - User-Agent, Client Hints, header order, TLS ClientHello, canvas/WebGL quirks, installed fonts, screen metrics. A real Chrome on real hardware produces a coherent fingerprint; a headless script or spoofed UA contradicts itself, claiming Chrome while missing the headers Chrome always sends, or claiming macOS while rendering like a Linux container. 
• Behavioral heuristics. The last line of defense watches how the visitor moves - mouse paths, click timing, scroll patterns, form-fill speed. Humans are biologically noisy; scripts are metronomic. It works until the automation learns to add noise of its own.

‍

Every one of these techniques rests on the same assumption - that machines move differently from humans, and that the difference is measurable. For thirty years, it was. That's no longer the case.

‍

The New Problem: AI Agentic Traffic

‍

The automation hitting a site today takes many forms. It could be a real person driving an AI browser, an AI agent fetching pages on behalf of a paying customer, a declared search crawler that drives discovery, or an uptime monitor the operator depends on. It could also be a credential stuffer, a carding bot, or a scraper run for profit. All of them share the same infrastructure, the same engines, the same fingerprints.

‍

One of the hardest of these to place is the AI browser. When a person opens ChatGPT Atlas, Perplexity Comet, or Dia and asks it to research a product, compare options, or complete a booking, the resulting requests are generated by an AI but emitted by a real, full Chromium browser running on the user's own machine. These are not headless emulations pretending to be a browser - they run the same rendering engine, the same network stack, the same JavaScript runtime as the Chrome on your own laptop. There is nothing fake to catch. Here is what that means for each layer of the stack:

‍

• Signature and fingerprint rules look for a tell - a known-bad User-Agent, or a self-contradiction across the headers, TLS handshake, and canvas output. An AI browser offers neither: it sends a genuine Chrome User-Agent and a perfectly consistent fingerprint, because it genuinely is Chrome.
• Behavioral heuristics look for movement that's too clean. But an AI agent driving a real browser scrolls, moves the pointer, and types with variation - and the human can take control between prompts, mixing the signals completely.
• Intent is the hardest layer to read. Even a perfect fingerprint on the traffic doesn't tell you the goal. A competitor's agent scraping your catalog to undercut your prices and a customer's agent comparing options before it buys send identical requests - same engine, same behavior, opposite intent.

‍

The line between detectable and undetectable automation has been blurring for years. A 2020 study in Knowledge-Based Systems reported that roughly half of all web users were bots, and that three-quarters of the bad ones were "Advanced Persistent Bots." These bots ran on headless or automation frameworks and deliberately imitated the mouse movements and clicks of human navigation - making them, as the authors put it, "very hard to detect." AI browsers take this one step further - there's no imitation involved, because the browser is real.

‍

The security industry isn't the only field reaching this conclusion. A 2025 paper in the APA's Psychological Methods studied how to keep bots out of online research. The authors found that browser fingerprints alone "may prove ineffective," and that LLM-driven bots now defeat the CAPTCHAs, attention checks, and open-ended questions that used to filter them out.

‍

The hard question to answer is no longer whether a request is automated. Because increasingly, it is. The question is what the automation is there to do - and User-Agent strings, fingerprints, and behavioral signals say nothing about intent.

‍

What The Lab Found

‍

To pressure-test all of this we built a controlled lab - a realistic site instrumented with behavioral telemetry, honeypots, and other tripwires, running both locally and on a public edge deployment. We established baselines with mainstream browsers, then ran several classes of automation against the same site - scripted clients (curl, python-requests), headless frameworks (Playwright, Puppeteer), AI agents and browsers, and automated scraping tools. We compared each one request by request, from HTTP headers down to the TLS handshake and into in-page behavior. Here's what we found, by category:

‍

• Command-line tools and scripts are trivially detectable. They send short, honest product User-Agents and skip the browser ritual entirely - no Client Hints, no Fetch-metadata, no JavaScript execution.
• Headless automation frameworks are still detectable, and they leak identifying markers in multiple places. The HeadlessChrome token appears in the Client Hints header as well as the User-Agent, and the "Google Chrome" brand that real Chrome always sends is missing. When a script runs, navigator.webdriver returns true, and the interaction trail is empty - no pointer movement, no clicks. No human session looks like that. 
• AI-driven browsers are the hard case. Some were byte-for-byte identical to a real browser on the wire, and the ones that weren't differed only by a fragile quirk that a routine update could erase at any time. There's no header- or fingerprint-level rule that reliably separates them from a human driving the same browser - not without an unacceptable false-positive rate against real customers.
• Self-identifying agents are detectable only because they choose to be. They name themselves in the request, and some behave distinctively - politely checking what they're allowed to crawl, or originating from recognizable cloud ranges. The signal is clean where it exists, but it's entirely voluntary, and a self-declared name is a label, not proof.

‍

Taken together, no single passive signal - User-Agent, headers, TLS, network origin, or behavior - holds up on its own against the agents that matter most, and combining them doesn't change that. We instrumented the lab site to watch for the tells that automation usually misses - mouse paths (humans wander, scripts snap straight to the target) and keystroke timing (humans type with uneven rhythm, scripts at a near-constant cadence). Today's agents score within human range on both measures, and the margin narrows every time a new agent version ships. The only automation we can reliably catch is the kind that isn't trying to hide - crude scripts that announce themselves, and agents that voluntarily identify. Everything else already looks human.

‍

<aside>

This forces a reframe: visibility before enforcement. The right question is no longer "is this a bot?" It's "how much of my traffic is agent-driven, which agents are responsible, what are they doing, and how is my edge treating them right now?" Operators have decades of tooling to understand traditional bots and almost none to understand agents. The opportunity is to build that same situational awareness for the agentic world.

‍

</aside>

‍

The Tension and What You Can Actually Do About It

‍

The lab results confirm what the research already suggested: the old detect-and-block model no longer works:

‍

• You can't block it because agent traffic is becoming demand. AEO (Answer Engine Optimization) means automated traffic needs to scrape your content for your brand to show up in AI-generated answers. AI agents with purchasing capabilities need access to your checkout to complete transactions on behalf of real customers. Block all of it and you disappear from the layer through which people are starting to discover and buy. And the volume is far from marginal - the 2026 Thales (Imperva) Bad Bot Report found that automated traffic now accounts for 53% of all internet traffic, with AI-driven bot activity rising more than tenfold in twelve months.

‍

• You can't just allow all of it, though. The same thing that makes a legitimate shopping agent indistinguishable from a human - a real browser, real fingerprints, real behavior -also gives fraud a perfect disguise. HUMAN Security documented an AI browser-agent running a carding workflow against a live checkout: eleven card-add attempts and six payment attempts across two sessions, then a pivot to loyalty-point redemption when the card paths failed. The same report found that across every interaction analyzed, just half a percent separates harmless automation from malicious. At the network level, good agents and bad ones are virtually identical.

‍

What You Can Do About it Today

‍

Reliably detecting anonymous agents is an open research problem, and the agents themselves change with every release - so the pragmatic move isn't to wait for a perfect detector. It's to take action fast on what we can resolve today and keep adapting as the landscape shifts. Two moves are practical right now:

‍

1. Lean on authenticated agents

‍

A self-reported name is a name tag, not an ID - and the gap is actively abused. HUMAN's Satori threat-intelligence team found that a significant share of requests claiming to be ChatGPT, Mistral, or Perplexity bots did not actually originate from those operators' infrastructure. Attackers spoof trusted crawler User-Agents to slip past robots.txt allowlists and rate-limit exemptions. To act on agent identity - to allow, meter, or challenge based on who an agent claims to be - you need proof, not a string.

‍

That proof is arriving fast. The industry is converging on cryptographic agent identity - the agent signs its requests with a key only it holds, and the origin verifies that signature against a published public key. Once identity is cryptographic, spoofing a User-Agent string no longer gets you in. Two approaches are already in production or close to it:

‍

• Web Bot Auth is an IETF effort championed by Cloudflare, built on HTTP Message Signatures (RFC 9421). It works the way you'd expect: the agent signs its requests, the origin checks the signature against the operator's published public key, and a spoofed User-Agent can't fake that. It's already in production - Cloudflare folded Message Signatures into its Verified Bots program and launched a Signed Agents classification, and AWS WAF added support in late 2025. Adopters include OpenAI, Vercel, Shopify, and Visa.

‍

• Token-based agent identity works differently - instead of signing each request, the agent carries a short-lived token with its identity claims baked in. The origin verifies those claims against the issuer's public keys. Several competing IETF drafts are in play. One backed by AWS and Ping Identity, gives a good sense of where this is heading - think OAuth/OIDC, built for machines.

‍

2. Monetize, don't just gate

‍

Once you can prove which agent is knocking, you don't have to choose between allow and block - you can charge. Return an HTTP 402 Payment Required, quote a price, and let the agent pay for access rather than get turned away. Cloudflare's Pay Per Crawl is an early example of exactly this. The scraper that used to drain your bandwidth for free now pays per request, and the shopping agent that brings real demand gets a clean, metered lane in. This only works on traffic you can identify, which is why it rides on top of the authentication layer above.

‍

Both approaches help, and neither is a complete answer. An anonymous AI driving a real browser from a residential IP is, for now, indistinguishable from a human - and pretending otherwise just adds false positives that hit real customers. What authenticated identity and monetization do is let you convert the part of the problem you can solve today into value, and put you in a better position as the signed-agent ecosystem grows. With the major cloud, commerce, and model vendors already participating, that's happening fast.

‍

The stakes only go up from here. Automated requests already make up roughly 57% of the traffic to actual web pages (Cloudflare Radar), and Cloudflare projects bots will outweigh humans across all internet traffic by 2027. The math is simple - a single agent completing one task can visit a thousand times the pages a person would. The traffic we've shown you can't reliably tell apart is already the majority of what hits your site, and it's still climbing. Every wrong call - blocking real demand or letting through fraud - gets more expensive.

‍

The Bottom Line

‍

The bot problem didn't change. Scalping, fraud, scraping, ad abuse, disruption - these are the same economic attacks that drove CAPTCHAs and heuristics thirty years ago. What changed is that the abuse now looks exactly like legitimate demand. We're not going to block our way out of that, and we can't yet reliably detect the anonymous version of it. 

‍

But "can't block, can't perfectly detect" does not mean "do nothing." The place to start is understanding what's actually hitting your site - which agents, doing what, at what volume - and then acting on the ones willing to prove who they are through cryptographic identity. The operators who get there first will be the ones who get to decide what to do about the rest.

‍