Techniques and signal families

How automated-traffic detection actually works — the signal families, the coherence model that ties them together, and the evasion arms race — read as mechanism, with each claim’s evidential weight marked.

What this section claims

  • Detection rests on no single signal. It works by combining families of signals — network/protocol, browser fingerprint, behaviour, session — and checking them for coherence: whether a client is internally consistent with what it claims to be.
  • Evasion is the mirror image: the practical problem is aligning every layer at once, because a mismatch in any one layer is itself a detection signal. Defender documentation and scraper-side guides converge on this framing independently — which is the strongest reason to believe it describes the real mechanism.
  • Each signal family is individually defeatable and individually noisy. Its value is cumulative and probabilistic, which is why production systems fuse families into a score rather than applying a single binary rule.
  • The evidential position is uneven and worth stating up front: the mechanisms are well-documented and controlled studies show the signals work in narrow settings, but production efficacy against an adaptive adversary is mostly asserted by vendors, not independently measured.

The layered model: coherence and alignment

The useful mental model is not “a list of detection tricks.” It is a stack of layers — network and IP, TLS handshake, HTTP protocol and headers, browser/JavaScript environment, session state, and on-page behaviour — each of which exposes signals, and a detector that asks whether those signals are mutually consistent with a single, plausible human-operated browser.

The clearest concrete instance is a coherence rule rather than a blocklist: Cloudflare documents detecting when a client “sends headers in a different order than the browser it claims to be” (Cloudflare, Detection IDs). Nothing about a single header is malicious; the signal is the mismatch between the claimed identity (a User-Agent saying “Chrome”) and a low-level property (header order) that the real Chrome would produce differently.

What makes this more than a vendor assertion is that the scraper side describes the same structure from the opposite direction. Public bypass writeups for named commercial systems frame the task as multi-layer alignment and state plainly that an attempt fails when only one layer is patched, because detectors look for cross-layer mismatch rather than single bad fields (ScrapingBee, PerimeterX bypass; ScrapFly, Imperva bypass; niespodd). Two adversarial sources — the people selling detection and the people selling evasion — independently arriving at “coherence across layers is the game” is about as much corroboration as this opaque field offers. The rest of this section walks the layers, then returns to how they are fused.

Network and protocol signals (TLS/JA4)

Below the application layer, the TLS handshake itself carries a fingerprint. The ordered set of properties a client offers in its ClientHello — protocol and version, cipher list, extensions, ALPN — is summarised by schemes such as JA3 and JA4, and a non-browser HTTP client typically produces a handshake that does not match any mainstream browser’s (Jarad & Bıçakcı 2026). The appeal is that this sits beneath the easily-changed fields: spoofing a User-Agent string or rotating IPs does not, by itself, alter the underlying TLS stack.

The evidence that this classifies well is real but bounded. Gradient-boosted models on JA4-derived records report an AUC around 0.998 (Jarad & Bıçakcı 2026) — a figure that should be read with care, because the study’s “bad bot” label is derived partly from the same application field that also feeds the model as a feature, so the headline number may partly reflect dataset-metadata separation rather than transferable TLS-based detection. The more durable point is the structural one the paper is honest about: TLS fingerprinting is strong against simple non-browser stacks and superficial spoofing, and weak against full-stack browser automation that uses a real browser’s network stack. It catches scripts that aren’t browsers; it does not catch a browser being driven. Scraper-side accounts agree that JA3/JA4 consistency with the claimed browser is something evasion has to satisfy (ScrapFly, Imperva bypass).

Browser fingerprinting

In the browser, the surface widens enormously. Browser fingerprinting combines many independently-weak attributes — User-Agent and headers, language, timezone, screen and platform, fonts, plugins, and JavaScript-exposed device signals such as Canvas rendering, WebGL vendor/renderer, AudioContext, and hardware concurrency — into a combined identifier, without storing anything on the client (Laperdrix et al. 2020). The foundational correction this literature offers is that fingerprinting is not one signal or a cookie replacement; it is the combination, evaluated with entropy and anonymity-set measures, that produces identifiability.

Two cautions matter for reading any fingerprinting claim. First, uniqueness is population- and time-dependent: volunteer studies reported very high uniqueness (Panopticlick ~83.6%; AmIUnique ~89% desktop), but a much larger general-population study (Gómez-Boix et al.) found substantially lower uniqueness, and a recent US dataset put combined-fingerprint uniqueness at roughly 60% (Berke et al. 2025). A number quoted without its population is close to meaningless. Second, the same attributes that aid detection also leak demographic information and carry unequal privacy cost across groups (Berke et al. 2025) — the signal families used to catch bots are not privacy-neutral, which is a real constraint on deploying them, not an aside.

A useful distinction to hold separate, because it is easy to conflate: detecting fingerprinting scripts, using fingerprints to detect bots, and bots evading fingerprinting are three different problems. FP-Inspector addresses only the first — it detects fingerprinting scripts via static and dynamic JavaScript analysis at ~99.9% accuracy and finds such scripts on roughly 10–30% of top sites (Iqbal et al. 2021). That high number is about finding fingerprinting code, not about catching bots; importing it as bot-detection performance would be a category error. The same work is useful here for a second reason: it quantifies the cost of countermeasures — blanket API restriction broke around 68% of tested sites versus around 30% for targeted restriction (Iqbal et al. 2021) — which is why neither defenders nor browsers can simply switch the fingerprinting surface off.

Behavioural signals and their evasion

If fingerprints describe what a client is, behavioural signals describe what it does: request timing and navigation patterns in server logs, and — at finer grain — mouse and touch dynamics. The behavioural layer is attractive because it is harder to fake convincingly than a static attribute: changing a header is trivial, while reproducing human neuromotor movement is not.

Mouse dynamics can separate humans from synthetic movement with high accuracy in controlled tasks — around 93% from a single trajectory and up to ~98.7% with combined features, using neuromotor (Sigma-Lognormal) modelling (Acien et al. 2021). But the framing distance is large: that result is a constrained point-and-click task, not natural browsing, and the “bots” are trajectory generators rather than full automation stacks. The web-log side is anchored by the same caveat seen elsewhere — request-pattern features detect simple bots almost perfectly but advanced ones poorly (Iliou et al. 2019; Iliou 2022).

Behaviour is also where the arms race is clearest, and where the honest reading is “adaptive game, not solved problem.” Reinforcement-learning agents can learn browsing behaviour that evades web-log detectors; generative models can synthesise mouse trajectories that reduce a CNN detector’s recall to around 0.45 in the web setting (Iliou 2022). Crucially, the same work shows retraining the detector on the evasive behaviour closes the gap — after which further adversary training reopens it. Detection here is not a static classifier you deploy once; it is a moving equilibrium. And these are capability demonstrations on researcher-generated bots, not evidence that RL- or GAN-driven evasion is common in production — a distinction the studies themselves draw.

Coherence checks and scoring

The layers above are combined in two ways. The first is explicit coherence rules — the header-order-versus-claimed-browser check is the documented example, exposed to operators as discrete detection identifiers they can log and act on (Cloudflare, Detection IDs). Cloudflare describes these as detecting predictable bot behaviour “with no overlap with human traffic,” which is the vendor’s claim; the documentation gives the mechanism, not a false-positive rate, so it should be read as capability rather than measured performance.

The second is probabilistic fusion. Cloudflare documents running multiple engines because simple and sophisticated bots need different strategies: a heuristics engine matching known fingerprints, JavaScript detections that catch headless browsers via lightweight client-side injection, and a machine-learning engine that maps request features, headers, session characteristics, and browser signals to a 1–99 bot score (Cloudflare, bot detection engines). This mirrors the academic ensembles (Iliou 2022) and is the practical reason no single signal “decides”: each family contributes a probability, and the score is the fusion.

The strongest independent anchor that this fusion catches real evasive traffic is FP-Inconsistent (Venugopalan et al. 2025), which ran purchased evasive bot traffic against commercial detection on a honey site and found fingerprint inconsistencies that gave the bots away. It is the best non-vendor evidence here — with the standing limit that its threat model is impression fraud on a single honey site, so it speaks to one setting rather than production breadth.

Challenge-response: CAPTCHA and proof-of-work

Challenge-response systems are a distinct layer: rather than passively scoring, they actively require the client to do something. Modern challenges increasingly run invisibly — Cloudflare’s Turnstile operates in managed/invisible, non-interactive, and interactive modes, and a pure HTTP client fails because it cannot execute the JavaScript or produce the required challenge token (Cloudflare, Turnstile; ScrapFly, Turnstile bypass). reCAPTCHA v3 went further, replacing the visible puzzle with a background behavioural risk score.

The instructive evasion result is Akrout et al. (2019), who trained a reinforcement-learning agent to move the mouse so as to obtain a high reCAPTCHA v3 score, reporting ~97% success. Two details matter more than the headline. First, their Selenium attempt was detected and scored poorly, so they drove a normal browser with an external mouse-control tool — a concrete demonstration that automation-framework artefacts are themselves a strong signal. Second, this is one reCAPTCHA setup in 2019; it is exactly the kind of narrow, high-percentage result that should not be generalised to “reCAPTCHA is broken” today. The broader point for the page is that behavioural risk scoring becomes an adversarial learning target the moment it is deployed, and that CAPTCHA-solving-as-a-service exists as a friction-reduction layer rather than a wall.

Where this sits: bot detection as ML-based detection

It is worth stepping back from the specific signals to note that none of this is peculiar to bots. Bot detection is one application of machine-learning-based cyber detection, and it inherits that field’s structure: supervised models, which classify against known labels and are strong on patterns they were trained on but weak on anything outside that distribution; and unsupervised / anomaly detection and behavioural-baseline (UEBA-style) methods, which learn a model of “normal” for a user, host, or traffic stream and flag deviations without needing labels — useful for novel or low-frequency behaviour, but at the cost of false positives and a dependence on how good the baseline and features are (Kolhar & Sridevi 2026, a conceptual overview used here only to situate the field, not as evidence). The in-scope instances on this page sit on both sides: the supervised ensembles and CNNs of the academic work (Iliou 2022), and the vendor anomaly/heuristics engines that flag outliers against learned norms (Cloudflare, bot detection engines).

Seeing it this way explains why the same caveats keep recurring above rather than looking like repeated hedging. The constraints are general properties of ML detection, not bot quirks. A supervised model is only as good as its labels — which is why proxy labels (Iliou et al. 2019) and label leakage (Jarad & Bıçakcı 2026) keep undercutting headline scores. An anomaly detector is only as good as its baseline and features, and an adversary who learns the baseline evades it — the same retrain-then-re-evade loop seen in the behavioural section. And both must be tuned against a false-positive cost, because misclassifying real users is expensive, which is why production systems run at deliberately low false-positive working points and fuse signals into a probability rather than deciding on one. So the useful question is not “can ML detect bots?” but: what signals are available, which labels are trustworthy, how does the model fail, and how are its decisions reviewed.

That last clause matters because detection in practice is a workflow, not a model in isolation: an anomaly model raises an alert, that alert is triaged with behavioural context, a human makes the decision, and the outcome feeds back into retraining (Kolhar & Sridevi 2026). UEBA-style behavioural baselining is most relevant to this project where it overlaps with account abuse — unusual access patterns as a signal for credential stuffing or account takeover — rather than as a general security-operations topic. The honest reading is the same one the rest of the page reaches by other routes: ML moves the cost and shifts the equilibrium of the arms race; it does not end it.

What this section defers, and a note on evidential weight

Three things are deliberately out of scope here. Graph and entity-resolution signals — cross-session and cross-account linking — are a genuine production family, but the project has almost no public evidence on them, so they are deferred rather than guessed at. The AI-agent shift as territory belongs to Background; within Techniques, AI-agent detectability is simply an application of the families above — FP-Agent (Wang et al. 2026) finds AI browsing agents distinguishable from humans on browser and behavioural fingerprints in a controlled honey-site setting, which is the cleanest non-vendor anchor for where agents currently sit, and explicitly a benign-task, closed-world, point-in-time result rather than evidence about adversarial humanisation or prevalence. Operational bypass detail is excluded by design: the scraper-side sources are used only at the level of which signal families exist and why coherence matters, never as alignment recipes. Project-run reproductions of any of this belong in Methodology; this section reports others’ experiments as evidence of how the mechanisms behave.

On evidential weight, the pattern across every family above is the same. Vendor documentation gives method, not efficacy (capability). Academic results are measured, but in controlled or constrained settings whose framing distance from production is large. The one genuinely independent operational anchor is a single honey site. So the mechanisms are well-evidenced and the signal families are real; what remains largely unmeasured in public is how well any of this works against a competent, adapting adversary at production scale. Read every performance number on this page as “in this setting,” not “in general.”

References

Sources are cited above by author/organisation and year; full details below. These are the works cited on this page, not the full register.

  • Acien et al. 2021 — Acien, A., Morales, A., Fierrez, J., & Vera-Rodriguez, R. (2021). BeCAPTCHA-Mouse: Synthetic Mouse Trajectories and Improved Bot Detection. arXiv:2005.00890.
  • Akrout et al. 2019 — Akrout, I., Feriani, A., & Akrout, M. (2019). Hacking Google reCAPTCHA v3 using Reinforcement Learning. arXiv:1903.01003. Proof-of-concept; one 2019 setup.
  • Berke et al. 2025 — Berke, A., et al. (2025). How Unique is Whose Web Browser? The Role of Demographics in Browser Fingerprinting among US Users. Proceedings on Privacy Enhancing Technologies 2025(1), 720–758. US Prolific sample, Dec 2023.
  • Cloudflare, bot detection engines — Cloudflare (2026). Bot detection engines (bot solutions docs). Vendor method description; not performance evidence.
  • Cloudflare, Detection IDs — Cloudflare (2026). Detection IDs (bot solutions docs). Vendor capability doc; no false-positive metrics.
  • Cloudflare, Turnstile — Cloudflare (2026). Turnstile — Overview (docs).
  • Iliou et al. 2019 — Iliou, C., et al. (2019). Towards a framework for detecting advanced web bots. ARES 2019. Simple-bot AUC 1.00 vs advanced-bot 0.64 on real web logs (proxy labels).
  • Iliou 2022 — Iliou, C. (2022). Machine Learning Based Detection and Evasion Techniques for Advanced Web Bots. PhD thesis, Bournemouth University. Web-log + mouse-CNN fusion; RL and GAN evasion (web mouse recall ≈ 0.45).
  • Iqbal et al. 2021 — Iqbal, U., Englehardt, S., & Shafiq, Z. (2021). Fingerprinting the Fingerprinters: Learning to Detect Browser Fingerprinting Behaviors (FP-Inspector). IEEE S&P 2021. Detects fingerprinting scripts, not bots.
  • Jarad & Bıçakcı 2026 — Jarad, G., & Bıçakcı, K. (2026). When Handshakes Tell the Truth: Detecting Web Bad Bots via TLS Fingerprints (arXiv:2602.09606). Preprint; headline AUC ≈ 0.998 carries a label-leakage caveat.
  • Kolhar & Sridevi 2026 — Kolhar, A., & Sridevi (2026). Secure and Ethically Aligned AI Solutions for Cybersecurity and Threat Detection (Ch. 11), IGI Global; with Future Trends and Innovation in Machine Intelligence for Cyber Risk Management (2025, Ch. 18). Secondary conceptual overview of ML in cybersecurity; admitted as non-load-bearing positioning background only (see scope note), not bot-specific or empirical evidence.
  • Laperdrix et al. 2020 — Laperdrix, P., Bielova, N., Baudry, B., & Avoine, G. (2020). Browser Fingerprinting: A survey. ACM Transactions on the Web 14(2), Article 8. doi:10.1145/3386040.
  • niespodd — niespodd (n.d./ongoing). Avoiding bot detection: How to scrape the web without getting blocked? (browser-fingerprinting). Scraper-side, dual-use; maintainer claims, not independent measurement.
  • ScrapFly, Imperva bypass — ScrapFly / Alisauskas, B. (2026). How to Bypass Imperva Incapsula when Web Scraping in 2026. Scraper-side, dual-use.
  • ScrapFly, Turnstile bypass — ScrapFly / Hisham (2026). How to Bypass Cloudflare Turnstile. Scraper-side, dual-use.
  • ScrapingBee, PerimeterX bypass — ScrapingBee / Krukowski (2026). How To Bypass PerimeterX Anti-Bot Protection System In 2026. Scraper-side, dual-use.
  • Venugopalan et al. 2025 — Venugopalan, et al. (2025). FP-Inconsistent: Detecting Evasive Bots using Browser Fingerprint Inconsistencies (arXiv:2406.07647). Purchased evasive traffic on a honey site; impression-fraud threat model.
  • Wang et al. 2026, FP-Agent — Wang, Shafiq & Vekaria (2026). FP-Agent: Fingerprinting AI Browsing Agents. Controlled honey-site measurement; benign-task, closed-world.