Bot scraping is surging, but blanket CAPTCHAs hurt real users. Here’s a clever micro-rule: challenge requests only when the URL’s query string includes an ampersand (&).
Credit: Simon Willison’s note on “CAPTCHA on at least one ampersand” inspired this lightweight mitigation idea.
What this rule does
Many scrapers hit URLs with multiple parameters—think ?q=...&page=...&sort=.... Requiring a quick challenge only when you detect an & adds targeted friction where bots congregate, while keeping most normal clicks friction-free.
When it works best
- Search, listing, and filter-heavy pages that invite parameterized crawling.
- Download endpoints (CSV/JSON) and internal APIs exposed over the web.
- Sites seeing “long-tail” automated queries with
page=,sort=,filter=, or tracking params chained by&.
How to implement (Cloudflare WAF example)
Create a Custom Rule that challenges when the query contains an ampersand and the request isn’t from a verified bot or a logged-in user. Expression sketch:
(http.request.uri.query contains "&") and not cf.client.bot and not (http.cookie contains "session=")
- Action: Challenge (Managed Challenge/Turnstile), not Block.
- Scope: Apply to high-risk paths only (e.g.,
/search,/api/,/export). - Observe first: Enable logging/“simulate” to tune exceptions before enforcing.
Docs: Cloudflare WAF Custom Rules. The same pattern applies to other WAFs/gateways with query matching.
Safeguards to reduce friction
- Exclude verified search engine crawlers (bot lists and IP verification).
- Allowlist logged-in users, admins, and API keys.
- Limit to GET requests or specific endpoints; keep auth flows untouched.
- Pair with rate limits for repeat offenders after a passed challenge.
Limitations
- False positives: legitimate users may share complex URLs with multiple params.
- Adaptive bots can solve some challenges—this won’t stop dedicated actors alone.
- Tune per country/ASN/device class to avoid penalizing mobile networks.
For broader context on bot threats and countermeasures, see the OWASP Automated Threats project.
Takeaway
Small, targeted friction (like an ampersand-triggered challenge) often yields 80% of the benefit with 20% of the user pain. Start narrow, measure, then iterate.
Enjoyed this nugget? Get more practical AI and security tactics each week—subscribe to The AI Nuggets newsletter.
