Cloudflare’s one-click toggle to block AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Bytespider, Amazonbot, and a long tail of named bots — has been in the dashboard since July 2024 (Security > Bots > “AI Scrapers and Crawlers”). It is free for every customer, the bot list is updated automatically as Cloudflare fingerprints new crawlers, and Cloudflare has said more than a million customers have enabled it. In 2025 the company layered pay-per-crawl monetization on top of the same control surface.
The feature is mature. The question for technical SEO leads is no longer “can I block?” — it is “should I?”
The pattern that decides it
Adoption skews by revenue model. Anecdotally, news and editorial publishers — sites that derive value from people reading their content — lean toward blocking. Sites that derive value from people acting on their content (transacting, filling forms, clicking affiliate links) lean toward allowing.
That is the whole decision in one line: block when the content itself is the product; allow when the content is the funnel.
What it can cost you to block
The real cost of blocking is lost discovery in AI search. If your audience increasingly asks ChatGPT, Perplexity, or Google’s AI Overviews a question your content answers, a blocked crawler means you are not in the answer — and not in the citation that drives the click.
We do not have a clean public number for “average citation share lost after enabling the toggle,” and you should be skeptical of anyone who quotes one with precision: citation share is noisy, query-dependent, and varies wildly by vertical. The honest framing is directional, not decimal — blocking removes you from a referral channel that is small today but growing for most publishers.
When blocking makes sense
Three scenarios where the block is defensible:
- You sell training data. If your archive is or could be a licensable corpus, free crawling is a leak. The New York Times, Reddit, and now Bloomberg have all monetized this.
- Your content is genuinely irreplaceable. Original investigative reporting, proprietary research, anything where a citation back to you isn’t substitutable for a citation to a paraphrase of you.
- You have a robust direct-traffic engine. If your homepage gets 70% of your traffic, AI citation referral is a smaller percentage of your numbers anyway.
When blocking is a mistake
- You’re a destination site competing for transactional intent. If users go to AI search to ask “what’s the best running shoe”, and you sell running shoes, being unciteable is being invisible.
- Your editorial team is investing in GEO. Blocking the crawler invalidates the work.
- You make money from being found. Affiliate sites, comparison sites, lead-gen sites: being a citation IS the funnel.
What we’re recommending
For most Smarter Ranking readers — mid-market brands and editorial teams — the answer is don’t block. Giving up AI-search discoverability is not worth it when you have no plan to monetize the archive you would be protecting.
The exception: if you have a clear monetization plan for your archive (a content licensing deal, a paywall expansion, an internal AI product, or Cloudflare’s own pay-per-crawl), then yes, the block makes sense.
For everyone else, leave the toggle off and put the energy into being more citeable, not less crawlable.