Skip to content
Anti-Bot

What Is Anti-Bot Detection?

Pim

Pim · Scrappey Research

May 31, 2026 5 min read

What Is Anti-Bot Detection? — conceptual illustration
On this page

Anti-bot detection is the set of techniques websites use to tell automated traffic apart from real human visitors — and then block, challenge, or slow down the automated half. Instead of relying on one clue, it stacks several together: IP reputation (whether an address has a history of abuse), browser fingerprinting (identifying details your browser leaks), TLS analysis (TLS is the encryption layer behind https), behavioral signals like how you move and click, and machine-learning scoring. The result is a risk score attached to every request. Cloudflare, DataDome, PerimeterX, Akamai, and Imperva are the dominant vendors, and most large sites use at least one of them.

Quick facts

Common vendorsCloudflare, DataDome, PerimeterX (HUMAN), Akamai, Imperva
Signal categoriesIP, TLS, HTTP, browser fingerprint, behavior, history
Action takenAllow, challenge (CAPTCHA), throttle, block
Detection layerFour layers: network → JS → WASM → behavioural

The four layers of anti-bot detection

Modern bot-protection products check every request against four separate layers. Failing any one layer is usually enough to get blocked — the layers act like a row of gates, not a points total. You have to clear all of them or you get nothing through.

LayerWhat's inspectedFires before…
1. NetworkTLS Client Hello (JA4), HTTP/2 SETTINGS frame, TCP options, IP reputation, ASNHTML is served
2. JavaScriptCanvas / WebGL / AudioContext fingerprints, navigator properties, Function.toString() inspection, extension probesXHR / API calls fire
3. WebAssemblyWASM SIMD CPU profile, SharedArrayBuffer timer precision, hyphenation dictionary checksChallenge token is issued
4. BehaviouralMouse movement Bezier curves, scroll cadence, keypress timing, click-to-event latencyScore is finalised over multiple requests

Each layer runs at a different moment. Layer 1 inspects the raw connection — including the TLS Client Hello, the first handshake message a browser sends, summarised as a JA4 fingerprint — before any HTML is even sent back. Layer 2 runs JavaScript in the page to probe the browser itself. Layer 3 leans on WebAssembly (compiled code that runs in the browser) for low-level CPU and timing checks. Layer 4 watches how you actually behave over several requests. So a scraper using curl_cffi (which only handles Layer 1) will pass against Layer 1-only vendors like older Imperva but fail against anything that loads sensor.js. A patched browser (Layers 1+2) will pass Akamai's static checks but fail DataDome's behavioural ML.

The five-vector coherence test

On top of the four detection layers, vendors run a separate identity-coherence check. The idea is simple: a real visitor's details should all tell the same story. These five vectors must agree:

  1. IP — geolocation, ASN type (residential / datacenter / mobile)
  2. TimezoneIntl.DateTimeFormat().resolvedOptions().timeZone
  3. Accept-Language — HTTP header
  4. WebRTC — candidate IP exposed by STUN/TURN
  5. DNS — resolver used (matches ISP or VPN?)

Here is what coherent looks like: an IP in São Paulo, a timezone of America/Sao_Paulo, an Accept-Language: pt-BR, a WebRTC candidate that matches the proxy, and a Brazilian ISP DNS resolver — every signal points to the same person in the same place. Now the giveaway: a US datacenter IP with a Tokyo timezone, English Accept-Language, and a WebRTC leak that reveals the operator's real home IP. That mismatch is the most common scraping signature and is trivially blocked. Proxy-rotation tools that change only the IP fail this test every time, because they leave the other four vectors pointing elsewhere.

What separates good detection from bad

Bad detection blocks on a User-Agent regex ("deny anything with `Bot` in the name") — trivially circumvented and it catches almost nothing real. Mediocre detection blocks datacenter IP ranges and JA3 hashes (a fingerprint of the TLS handshake) of known scraping libraries — this catches the lazy 80% of scrapers but misses anything running a real browser behind proxies. Good detection is the major commercial vendors. They pool signals from thousands of customer sites, update their models as new automation patterns appear, and correlate identities across requests — so passing one challenge is not permanent, because the same fingerprint is recognised on later requests. That is why detection accuracy is a continuously evolving field, not a fixed system that stays the same over time.

What detection systems weigh most

Three factors dominate. First, identity consistency: detection compares the IP type (residential, mobile, or datacenter), the browser environment, and whether the fingerprint matches configurations that actually exist in the wild. Second, behavioural realism: detection scores request timing, session-level cookie continuity, and whether request gaps look human or mechanically even. Third, model freshness: when a site adopts a new detection vendor, its scoring changes, which is why detection accuracy is best understood as an evolving capability rather than a fixed rule set. For authorized data collection on sites you own or are permitted to access, managed scraping APIs absorb this ongoing change so you do not maintain the configuration yourself.

Next in How automation gets detected · 2 of 6

Start with the most-checked category: is this browser headless?

What Is Headless Browser Detection?

Related terms

Concept map

Concept map

How Anti-Bot Detection connects

The terms most directly tied to this one. Hover a node to see its neighbours, click to preview, drag to rearrange.

0 terms · 0 connections
You are here · Anti-Bot
Building map…

Frequently asked questions

How do sites know I'm using a bot?

Usually it's a combination of signals working together — a datacenter IP, headless-browser leaks, a TLS fingerprint that doesn't match a real browser, unrealistic request timing — rather than any single giveaway. Modern systems score the whole picture, not individual clues.

Is anti-bot detection ever perfectly accurate?

No detection system is flawless, and none stays static. Vendors improve their models continuously, so the signals a given configuration produces are scored differently over time. Detection is best understood as an evolving capability rather than a fixed, solved system.

Which anti-bot vendor is the most aggressive?

DataDome and PerimeterX (HUMAN) tend to score automated traffic most aggressively. Cloudflare is everywhere and improving fast. Akamai is strong on financial and travel sites. Which one a given site uses depends on the target and changes over time.

Does respecting robots.txt affect bot detection?

Not directly — anti-bot vendors don't read robots.txt. Respecting it is good practice and lowers your legal risk, but the detection layer scores technical signals regardless of what robots.txt says.

Why is a request blocked even though my fingerprint passes a fingerprint-test site?

Fingerprint-test sites usually only check Layer 2 (JavaScript). Your block happened at Layer 1 (TLS / IP) or in the five-vector coherence test — neither of which the test site inspects. A quick tell: if you get a 403 before any JavaScript runs, the failure is at Layer 1.

Do all vendors run all four layers?

No. Imperva and AWS WAF default to Layer 1 plus a light Layer 2. Akamai, Cloudflare Bot Management, and PerimeterX run all four. DataDome leans heavily on Layer 4 behavioural ML. F5 Shape runs Layers 1 + 2 plus a custom VM that defies easy categorisation. The vendor cheatsheet entry maps which layers each one weights heaviest.

Last updated: 2026-05-31