Is Your Blog Blocked From AI Answers? A Crawler Checklist

Seven weeks from now, Cloudflare changes the default answer to a question you may never have been asked. On September 15, 2026, the service that handles edge routing for millions of sites will begin blocking AI training and agent crawlers by default on pages that display ads—and retiring the old one-click "Block AI bots" toggle entirely. If you're on Cloudflare's Free plan and you've never touched Security → Settings, you may wake up to discover your posts are no longer reaching the crawlers that power ChatGPT search results, Perplexity citations, or Claude's real-time retrieval.

The core problem is not that site owners are making bad policy choices. It's that a network-layer default they never set is enforcing a policy that contradicts what their robots.txt says. Your robots.txt file is a preference, not enforcement—Cloudflare states this plainly in its own documentation. The converse is the part small sites miss: a permissive robots.txt does nothing if the edge blocks the request first. The crawler receives a 403, JavaScript challenge, CAPTCHA, rate limit, or empty response instead of your page. Because the block happens at the edge, it's invisible in your origin logs. Your access logs show nothing because the traffic never arrives.

Being cited by ChatGPT, Perplexity, and Google AI Overviews now depends on bot policy you may never have configured. This post walks you through the five-layer audit to check which AI user agents can actually reach your posts, the September 15 deadline, and what the data says about whether blocking is worth it at all.

Three Bots, Three Jobs, Three Decisions

The question "should I block AI crawlers?" is no longer binary. AI crawlers have split into at least three distinct jobs—training, search indexing, and live user-triggered fetch—and each has its own user agent that can be allowed or blocked independently. Blocking training crawlers does not remove you from AI answers. Blocking search and fetch crawlers does.

OpenAI runs four separate agents

OpenAI's own documentation lists these four:

• OAI-SearchBot — "used to surface websites in search results in ChatGPT's search features." OpenAI explicitly says "we recommend allowing OAI-SearchBot in your site's robots.txt file." This is the crawler that determines whether your posts appear as citations in ChatGPT answers.
• GPTBot — "used to crawl content that may be used in training our generative AI foundation models." This is training, not retrieval. Blocking it keeps your content out of future model weights but does not affect citation eligibility.
• ChatGPT-User — "used for certain user actions in ChatGPT and Custom GPTs." For this one, "robots.txt rules may not apply" because the action is user-initiated. It fires when a ChatGPT user asks a question that requires fetching a specific page in real time.
• OAI-AdsBot — validates pages submitted as ChatGPT ads.

Perplexity runs two

• PerplexityBot — "designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models." This is search indexing, not training.
• Perplexity-User — handles user-triggered fetches. Perplexity states explicitly: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." Perplexity publishes IP ranges at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json.

Anthropic runs three

• ClaudeBot — collects "web content that could potentially contribute to their training."
• Claude-User — fires "When individuals ask questions to Claude."
• Claude-SearchBot — "navigates the web to improve search result quality for users."

Google-Extended is not what most people think it is

Google's documentation defines Google-Extended as "a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini." It "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

Therefore blocking Google-Extended does not remove you from AI Overviews or AI Mode. Google states: "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary"—eligibility flows from ordinary Search indexing, controlled by Googlebot robots.txt rules and preview controls like nosnippet, data-nosnippet, max-snippet, and noindex.

If you want to opt out of Google's generative AI features, you now have a separate control: as of June 3, 2026, Google added a Search Console toggle (currently rolling out to a subset of site owners in the UK) that lets you "decide if they want their site to appear in and help ground responses in our generative AI Search features (like AI Overviews, AI Mode or AI Overviews in Discover)." Google is clear: "Sites that opt out will not receive traffic or impressions from our generative AI features."

Bing is the awkward one

Microsoft handles AI usage at the page level, not via a separate user agent token. Content tagged NOCACHE "may be included in Bing Chat answers. We will only display URL/Snippet/Title in the answer." Content tagged NOARCHIVE "will not be included in Bing Chat answers, not be linked to in the answers." Both still appear in ordinary Bing results, and if both tags are present, Bing treats it as NOCACHE.

What Cloudflare Is Doing to Your Zone on September 15, 2026

On July 1, 2026, Cloudflare announced a three-way AI traffic taxonomy and set new defaults that take effect September 15, 2026. On the same date, Cloudflare's old one-click "Block AI bots" toggle is retired. This is the most important near-term deadline in AI crawler policy, and it affects every Cloudflare customer including those on the Free plan.

The three behaviours

Cloudflare's new framework divides AI crawlers into three categories:

• Search — a crawler that "collects or indexes your content, so it can answer questions about it later." Examples: OAI-SearchBot, PerplexityBot, Claude-SearchBot.
• Agent — "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now." Examples given: "chat fetch bots (e.g., ChatGPT-User) and browser-use agents."
• Training — "a crawler taking your content to train or fine-tune a model," where "your data is permanently absorbed into the underlying architecture." Examples: GPTBot, ClaudeBot, Google-Extended.

The September 15 defaults

"On September 15, 2026, we'll be setting new defaults for each of these three classifications. For all new domains" onboarding to Cloudflare, "Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default."

TechCrunch reports the default block on ad-supported pages affects "new Cloudflare customers, newly created sites, and all free-tier accounts," though Cloudflare's own post only spells out "all new domains onboarding to Cloudflare." Attribute the wider free-tier claim to reporting, not to Cloudflare's own statement, until you check your own zone's settings.

Mixed-use crawlers are the trap

Cloudflare warns: "multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors," naming "Googlebot, Applebot, and BingBot" as examples. Cloudflare's changelog adds that crawlers combining Search and Training "will be affected by the new defaults to block Training."

This matters: if you intended to allow search indexing but block training, a mixed-use crawler gets blocked entirely. You cannot split the difference at the user-agent level when a single bot does both jobs.

Where to change it

The dashboard path is /:account/:zone/security/settings, available to all customers including Free. For each behaviour you can "block on all pages, block only on pages that display ads, or choose not to block." Customers can opt out of the new defaults any time before September 15, 2026.

The old toggle is being retired

Cloudflare's "Block AI bots" setting "is deprecated and will be retired on September 15, 2026," when the three-way Search/Agent/Training policies replace it. Notably, the old toggle already "excludes mixed-purpose bots that are used both for Training and for Search," so sites relying on it may have been blocking less than they thought.

Managed robots.txt prepends, it does not replace

Cloudflare's managed robots.txt feature is available on all plans and "will prepend our managed robots.txt before your existing robots.txt, combining both into a single response." The managed block disallows Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot, and meta-externalagent, and emits Content-Signal: search=yes, ai-train=no, use=reference.

Note that this managed list is training-only. It does not contain OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User, or Perplexity-User. A site owner who enabled managed robots.txt has opted out of training but kept citation eligibility—which is a reasonable default and worth recognizing as such, rather than framing it as a mistake.

Cloudflare is clear about the limits of robots.txt: "robots.txt compliance is voluntary… Some crawler operators may disregard your robots.txt directives and crawl your content regardless." In August 2025, Cloudflare found Perplexity "repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity," running an undeclared stealth crawler at 3–6 million daily requests on IPs outside its published ranges. Cloudflare delisted Perplexity as a verified bot and shipped detection signatures to all customers including Free.

Bot Fight Mode: The Free-Plan Landmine

If you're on Cloudflare's Free plan, there is one setting that overrides every other rule you write: Bot Fight Mode. Cloudflare's documentation states plainly: "You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules."

Bot Fight Mode operates in a separate evaluation pipeline where Skip, Bypass, and Allow actions have no effect. It applies to the entire domain with no endpoint restrictions, and it issues "computationally expensive challenges that force the requesting client to perform CPU-intensive calculations."

This is confirmed independently in the WAF docs: skip rules can bypass rate limiting rules, Managed Rules, and Super Bot Fight Mode rules, but "you cannot skip Bot Fight Mode (available on the Free plan)."

Cloudflare also warns Bot Fight Mode "may challenge API or mobile app traffic," and directs AI bot handling to a separate "Block AI bots" configuration—meaning Bot Fight Mode does not natively distinguish an AI search crawler from a scraper.

AI crawlers cannot solve JavaScript challenges. The bots that read your pages for ChatGPT, Claude, and Perplexity don't run JavaScript. They get handed the challenge and nothing else. If Bot Fight Mode is on, your permissive robots.txt is irrelevant—the crawler never reaches your origin to read it.

A widely-shared Cloudflare walkthrough recommends creating a Security → WAF → Custom rule with a user-agent expression, action Skip, ticking "WAF Managed Rules, Bot Fight Mode, and Block AI bots," plus a curl verification. This contradicts Cloudflare's own documentation. The correct move for a Free-plan reader is to turn Bot Fight Mode off (or upgrade to Super Bot Fight Mode, which is skippable) rather than try to write around it.

The Five-Layer Audit

This is the practical checklist to run on your own site. Ordered from cheapest to most involved, expect this to take 20–30 minutes on a first pass.

Layer 1: Policy (robots.txt)

Fetch your own /robots.txt in a browser and read the whole response, not the file in your repo. Cloudflare prepends its managed block if enabled, so the served file may differ from the committed one.

Check that OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, Googlebot, Bingbot, and Applebot are not caught by a Disallow: /—including via a User-agent: * group.

The classic error: a Disallow under one user-agent group does not apply to others, but a blanket User-agent: * followed by Disallow: / blocks everything unless explicitly overridden. If you've added specific Allow rules for AI search crawlers, they must come after any wildcard disallows in their own user-agent groups.

Layer 2: Edge / CDN

In Cloudflare: Security → Settings for the new Search/Agent/Training policies (all plans, since July 1, 2026); Security → Bots for the legacy "Block AI bots" toggle and Bot Fight Mode status.

On Free, if Bot Fight Mode is on, your only options are off or Super Bot Fight Mode. No custom rule can skip it.

On Vercel: Both relevant rulesets are off by default. Bot protection managed ruleset is labeled Off, and AI bots managed ruleset is labeled Allow when inactive. Confirm both are still at their defaults unless you've deliberately changed them. If you have switched Vercel's bot protection on, it "identifies clients that violate browser-like behavior and serves a javascript challenge to them"—which is exactly what a non-JS AI crawler fails. Vercel does automatically exclude verified bots from that ruleset.

Stacking Cloudflare in front of Vercel degrades bot detection: "Bot Protection doesn't work when a reverse proxy (e.g. Cloudflare, Azure, or other CDNs) is placed in front of your Vercel deployment," causing obscured detection signals and frequent re-challenges. This is a very common indie-dev topology—if you're running both, your edge behavior is governed by Cloudflare's rules, not Vercel's.

Layer 3: Rate limits

An AWS WAF case study shows blocks that only appear on the 4th or 5th request in a burst: the site served the full 136KB page for the first three requests, then returned "a stub with a JavaScript challenge and an HTTP 202" on requests 4 and 5. The site's robots.txt was fully permissive throughout. Rate limits, not policy, were the block.

A single curl test will pass; a crawl will not. If you're testing, send a short burst—five requests over ten seconds—not one request.

Layer 4: Evidence, not guesses

Cloudflare AI Crawl Control (available on all plans) → Crawlers shows which AI services actually reached your zone. The Directives tab shows robots.txt fetch health (successes vs 400+ responses) and a violations table with bot name, operator, path, and request counts, though "violations are not logged in real-time."

This is the closest thing to ground truth a free-plan user has, because origin logs won't show edge-terminated requests. If OAI-SearchBot appears in your violations table with disallowed paths and non-zero request counts, you have a real block to fix. If it doesn't appear at all, the block may be happening before the crawler identifies itself, or your content simply hasn't been indexed yet.

Layer 5: Rendering

curl your post URL and read the raw HTML. If your post body only appears after JavaScript runs, no major AI crawler except Google's will ever see it.

Vercel and MERJ's analysis covering OpenAI, Anthropic, Meta, ByteDance, and Perplexity bots found that "none of the major AI crawlers currently render JavaScript"—although they do fetch JS files without executing them. Google is the exception, because Gemini surfaces ride on Googlebot's rendering infrastructure. ChatGPT spent 11.50% of requests fetching JS, Claude 23.84%, but neither executes it.

For a static-site blog built with Astro, Next.js static export, or Hugo, this is usually fine—your HTML is already in the initial response. For a SaaS marketing site built as a single-page app where content renders client-side, it usually isn't.

Common mistakes worth naming

• Treating "block AI" as one decision instead of three (training / search / agent).
• Blocking Google-Extended believing it removes you from AI Overviews—it doesn't; it only affects Gemini Apps and Vertex AI training.
• Pasting a full ai.robots.txt blocklist (the open-source maintained list) without stripping the retrieval bots. The list is useful as a starting point, but copying it wholesale blocks OAI-SearchBot, PerplexityBot, and Claude-SearchBot along with the training crawlers.
• Trusting a spoofed-user-agent curl as proof, when the edge verifies by IP, reverse DNS, or cryptographic signature. Vercel documents three bot verification methods, including Web Bot Authentication through RFC 9421 HTTP Message Signatures—meaning a curl -A "GPTBot" test tells you about user-agent-string rules only, not IP-range allowlists or signature validation.
• Writing an llms.txt instead of fixing access. Google's AI optimization guide (last updated July 10, 2026) states outright: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search," and "Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." Google's John Mueller called llms.txt "purely speculative for now… the file has existed for years, yet none of the AI systems use it," and emphasized that ensuring sites don't block AI agents from accessing their content is "the biggest hurdle, for most sites."

The Honest Counter-Case: Should You Block After All?

The data on whether to allow AI crawlers at all is mixed, and the best evidence points in opposite directions depending on your site's profile.

The crawl-to-refer imbalance is real

Cloudflare measured crawl requests against referrals for the first week of August 2025, all industries: Anthropic (ClaudeBot) 50,000:1; OpenAI (GPTBot) 887:1; Perplexity 118:1. For News & Publications specifically: Anthropic 2,500:1; OpenAI 152:1; Perplexity 32.7:1.

Training traffic was "responsible for nearly 80% of the crawling from AI bots" in July 2025. Cloudflare's June 2026 report found that "52% of crawler requests are now for AI training as of June 2026, up from 22% in Spring 2025," and "over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages."

This is a bandwidth and fairness problem. Cloudflare had "over one million customers" choosing its AI crawler blocking option between September 2024 and July 2025.

But blocked sites still get cited—if they're already famous

BuzzStream research published March 19, 2026 (using Citation Labs' XOFU tool, 4 million citations from 3,600 prompts across 10 industries, focused on the top 50 news sites) found:

• 88.2% of sites blocking GPTBot still appeared in citations
• 82.4% of sites blocking OAI-SearchBot still appeared
• 92.3% of sites blocking Google-Extended still appeared
• "Roughly 70% of all ChatGPT citations in the dataset came from sites that block ChatGPT's retrieval bots," rising to "95% when looking at sites that block training bots"
• CNBC (blocking multiple bots) appeared 1,298 times; Yahoo (blocking Google-Extended) appeared close to 30,000 times

This study is real and honestly reported. It also does not transfer to a solo founder's SaaS blog. The top 50 news sites are cited because they are quoted, aggregated, syndicated, and summarized everywhere else on the web—the model does not need to fetch nytimes.com to know what the NYT reported. A 40-post indie blog has none of that secondary corpus.

The cloro.dev study (1,058 domains, published July 6, 2026) points the other way and is closer to the audience's situation: median citation propensity (ChatGPT citations per Google-organic appearance, two-day window) was 0.003 for GPTBot-blocking domains vs 0.417 for non-blockers. The authors explicitly caveat that findings are "directional—the corpus reflects cloro's monitored prompt mix rather than the whole web," and that causation isn't proven. But directionally: if you are not the New York Times, blocking the retrieval bots likely costs you citations.

73% of sites have technical barriers blocking AI crawler access

Otterly.ai's AI Citations Report 2026 (January–February 2026, 1m+ AI citations across ChatGPT, Perplexity, and Google AI Overviews) found that 73% of sites have technical barriers blocking AI crawler access. Named barriers: robots.txt blocks, CDN rules blocking non-browser user agents, and JavaScript rendering.

The practical risk for a small site is not that the owner made a bad policy decision. It's that a network-layer default they never set is enforcing a policy they didn't choose.

What the Numbers Say It's Worth

AI platforms drive roughly 0.15%–0.25% of total global internet traffic, versus organic search's ~48.5% (global, 2026). By volume, AI referral traffic is a rounding error.

But the intent density is different. Ahrefs' own analytics (trailing 30 days at time of publication) found that AI search was 0.5% of traffic but 12.1% of signups, a 23× higher conversion rate than traditional organic search. ChatGPT referral traffic converts at 7.1%, second only to paid search at 7.8%, per Similarweb's clickstream panel for April–May 2026.

Ahrefs includes its own caveat, which any honest post should carry: "users from AI search click links 75% less than they do in traditional organic search," and the author questions whether the conversion advantage would hold at higher volume.

Citation frequency of AI answers grew from 0.6% (January 2025) to 2.8% (August 2025). ChatGPT web visits grew 84% September 2024–March 2026; Gemini ~9×; Claude ~770% over the same period.

The honest summary: low volume, high intent, near-zero cost to stay eligible. The configuration you set today takes an hour. Staying accidentally blocked costs you compounding citation equity you'll never see in your analytics because the request was terminated at the edge.

The Recommended Default for a Small Blog

Allow Search and Agent. Block or throttle Training. Don't write an llms.txt. Do check Search Console's new generative-AI reports if you have access (currently rolling out in the UK).

Here's a paste-able robots.txt block that allows search and user-fetch crawlers while disallowing training:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: cohere-ai
Disallow: /

Then go check the edge. If you're on Cloudflare:

1. Security → Settings: confirm Search is allowed, and Training/Agent policies match your intent.
2. Security → Bots: confirm Bot Fight Mode is off, or upgrade to Super Bot Fight Mode and write the skip rule.
3. AI Crawl Control → Crawlers: within a week, check which bots are actually reaching your zone.

If you're on Vercel, confirm both managed rulesets are still at their defaults (Off / Allow) unless you've deliberately tuned them.

If you enabled Cloudflare's managed robots.txt, you've already opted out of training and stayed eligible for citations—check Security → Bots to confirm it's active and read your served /robots.txt to see the prepended block.

Recheck This in September

The settings you audit today have a scheduled expiry. On September 15, 2026, Cloudflare's new defaults take effect and the old "Block AI bots" toggle is retired. If you're on Cloudflare—especially if you're on the Free plan, and especially if your blog earns revenue from ads or affiliate links that Cloudflare might classify as "pages that display ads"—you should return to Security → Settings after September 15 and confirm the new defaults match your policy.

The question is no longer "should I block AI crawlers?" The question is: have you set a policy at every layer that can enforce one, or are you inheriting defaults chosen by your CDN, your host, and your firewall provider?

Your robots.txt is not the file that decides this. Your edge is. And seven weeks from now, your edge's answer changes.

---

For the small sites publishing via GitHub PR against a static-site repo—where robots.txt is in version control and the publish button is a merge—there's a parallel discipline worth noting: the same way a permissive robots.txt does nothing if the edge blocks the crawler, a well-cited post does nothing if the links inside it 404 when a reader (or an AI) follows them. Magic Share's link verification runs the same kind of audit one layer up: every cited URL in a draft is resolved before the post is saved, and a verification report ships attached. It's no good publishing posts that 403 for crawlers for the same reason it's no good citing sources that 404 for readers—both are invisible failures that compound over time. Vercel's data showing ChatGPT spending 34.82% of fetches on 404 pages makes the link-hygiene point concrete.

If you're running a blog that compounds when it's tended year-round rather than in bursts, AI citation eligibility is a standing configuration, not a one-time task. Google now measures generative-AI impressions separately in Search Console. The posts you publish this month may be cited six months from now by a user asking a question you didn't anticipate, through a retrieval bot you forgot you allowed. The harvest is delayed, but the decision to leave the gate open is made today.

This post is also part of a broader question: whether AI-assisted content is allowed at all under Google's scaled content abuse policy. That post covered whether Google penalizes AI-drafted posts; this one covers whether AI crawlers can reach them. Together they form the two gates: policy and access. Both have to be open for a post to compound.

FAQ

If I block GPTBot but allow OAI-SearchBot, will my content still be used for training?

Possibly, through indirect pathways. OpenAI's documentation distinguishes the two: GPTBot is "used to crawl content that may be used in training our generative AI foundation models," while OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features." Blocking GPTBot prevents direct crawling for training. But if your content is cited in other indexed sources, quoted in public forums, or included in datasets assembled by third parties, it may still reach training corpora indirectly. The block is real but not hermetic.

Does blocking training crawlers hurt my rankings in regular Google Search?

No. Google states explicitly that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Blocking training-only crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) has no direct effect on your visibility in traditional search results or in AI Overviews, which are governed by Googlebot's indexing and your Search Console generative-AI toggle.

I use Cloudflare's Free plan and Bot Fight Mode is on. Can I write a custom rule to allow AI search crawlers?

No. Cloudflare's documentation is unambiguous: "You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules." Bot Fight Mode operates in a separate evaluation pipeline where Skip, Bypass, and Allow actions have no effect. Your only options on the Free plan are to turn Bot Fight Mode off entirely, or upgrade to Super Bot Fight Mode (a paid feature), which is skippable by custom rules. The widely-circulated workaround of writing a Skip rule that ticks "Bot Fight Mode" does not work on Free—it contradicts Cloudflare's own docs.

My robots.txt allows all crawlers, but I don't see AI traffic in my logs. What's blocking them?

The most common causes, in order: (1) an edge or CDN challenge (Cloudflare Bot Fight Mode, Vercel bot protection, AWS WAF rate limits) that terminates the request before it reaches your origin—your logs show nothing because the traffic never arrives; (2) your content is rendered client-side by JavaScript, and AI crawlers (except Google's) don't execute JS; (3) rate limits that allow the first few requests but challenge subsequent ones, so a single curl test passes but a real crawl doesn't; (4) you haven't been indexed yet—if your blog is new or low-authority, crawlers may not have discovered your content. Run the five-layer audit in this post to narrow it down.

Does the September 15 Cloudflare change affect my site if I don't run ads?

Cloudflare's stated default is to block Training and Agent "on the pages that display ads" for new domains. If your blog genuinely has no ads, affiliate links, or monetization that Cloudflare might classify as ad-supported, you may be unaffected in practice. But "displays ads" is Cloudflare's classification, not your declaration—it's an automated detection, not a setting you control. The safe move: go to Security → Settings after September 15 and confirm the policies applied to your zone match your intent, rather than assume you're exempt. If you're on a newly-created site or onboarding to Cloudflare for the first time after that date, the defaults apply regardless, and you can opt out through the same dashboard.