Is Your Blog Blocked From AI Answers? A Crawler Checklist
On September 15, 2026, Cloudflare changed the default answer to a question you may never have been asked. New domains onboarding to the service now pick from AI bot presets, and the preset for ad-supported sites disallows AI training and blocks agent fetchers on pages that show ads. The same release changed what "Block" means for training, so it now also stops Googlebot, Bingbot and Applebot. Cloudflare is also deprecating its old one-click "Block AI Bots" option. Whatever you set before, check what your zone does now: your posts only reach the crawlers behind ChatGPT search results, Perplexity citations and Claude's real-time retrieval if every layer lets them through.
The core problem is rarely a bad policy choice. More often, a network-layer setting the owner never revisited is enforcing a policy that contradicts what their robots.txt says. Your robots.txt file is a preference, not enforcement—Cloudflare states this plainly in its own documentation. The converse is the part small sites miss: a permissive robots.txt does nothing if the edge blocks the request first. The crawler receives a 403, JavaScript challenge, CAPTCHA, rate limit, or empty response instead of your page. Because the block happens at the edge, it's invisible in your origin logs. Your access logs show nothing because the traffic never arrives.
Being cited by ChatGPT, Perplexity, and Google AI Overviews now depends on bot policy you may never have configured. This post walks you through the five-layer audit to check which AI user agents can reach your posts, what changed on September 15, and what the data says about whether blocking is worth it at all.
In short:
- Training, search and user-triggered fetch are three separate crawler jobs. Blocking training does not remove you from AI answers; blocking search and fetch does.
- Since September 15, 2026, Cloudflare's "Block" setting for training also stops Googlebot, Bingbot and Applebot. Use the new Disallow AI Training option if you want to refuse training but stay in search.
- On Cloudflare's Free plan, Bot Fight Mode overrides every rule you write. Check it, then confirm with AI Crawl Control which bots reach you.
Three Bots, Three Jobs, Three Decisions
The question "should I block AI crawlers?" is no longer binary. AI crawlers have split into at least three distinct jobs: training, search indexing, and live user-triggered fetch. Each has its own user agent that can be allowed or blocked independently. Blocking training crawlers does not remove you from AI answers. Blocking search and fetch crawlers does.
OpenAI runs four separate agents
OpenAI's own documentation lists these four:
- OAI-SearchBot — "used to surface websites in search results in ChatGPT's search features." OpenAI explicitly says "we recommend allowing OAI-SearchBot in your site's robots.txt file." This is the crawler that determines whether your posts appear as citations in ChatGPT answers.
- GPTBot — "used to crawl content that may be used in training our generative AI foundation models." This is training, not retrieval. Blocking it keeps your content out of future model weights but does not affect citation eligibility.
- ChatGPT-User — "used for certain user actions in ChatGPT and Custom GPTs." For this one, "robots.txt rules may not apply" because the action is user-initiated. It fires when a ChatGPT user asks a question that requires fetching a specific page in real time.
- OAI-AdsBot — validates pages submitted as ChatGPT ads.
Perplexity runs two
- PerplexityBot — "designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models." This is search indexing, not training.
- Perplexity-User — handles user-triggered fetches. Perplexity states explicitly: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." Perplexity publishes IP ranges at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json.
Anthropic runs three
- ClaudeBot — collects "web content that could potentially contribute to their training."
- Claude-User — fires "When individuals ask questions to Claude."
- Claude-SearchBot — "navigates the web to improve search result quality for users."
Google-Extended is not what most people think it is
Google's documentation defines Google-Extended as "a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini," and for grounding answers in Gemini Apps and Vertex AI. It "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
Therefore blocking Google-Extended does not remove you from AI Overviews or AI Mode. Google states: "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary"—eligibility flows from ordinary Search indexing, controlled by Googlebot robots.txt rules and preview controls like nosnippet, data-nosnippet, max-snippet, and noindex.
If you want to opt out of Google's generative AI features, you now have a separate control. On June 3, 2026, Google began testing a Search Console toggle with a subset of UK site owners, and on August 31, 2026 it said the control had rolled out to all websites worldwide. The toggle lets site owners "decide if they want their site to appear in and help ground responses in our generative AI Search features (like AI Overviews, AI Mode or AI Overviews in Discover)." Google is clear: "Sites that opt out will not receive traffic or impressions from our generative AI features."
Bing is the awkward one
Microsoft handles AI usage at the page level, not via a separate user agent token. Content tagged NOCACHE "may be included in Bing Chat answers. We will only display URL/Snippet/Title in the answer." Content tagged NOARCHIVE "will not be included in Bing Chat answers, not be linked to in the answers." Both still appear in ordinary Bing results, and if both tags are present, Bing treats it as NOCACHE. (Bing Chat is now Copilot, but these 2023 tag rules are still the published mechanism; Cloudflare reports Microsoft is building a robots.txt "no training" preference targeted for early 2027.)
What Cloudflare Changed on September 15, 2026
On July 1, 2026, Cloudflare announced a three-way AI traffic taxonomy and said it would set new defaults on September 15, 2026. Those changes are now live. They matter most for sites onboarding to Cloudflare after that date, and for anyone who picks a "Block" setting for training from now on.
The three behaviours
Cloudflare's new framework divides AI crawlers into three categories:
- Search — "any behavior that collects or indexes your content, so it can answer questions about it later." In practice that covers crawlers like OAI-SearchBot, PerplexityBot and Claude-SearchBot.
- Agent — "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now." Cloudflare's examples: "chat fetch bots (e.g., ChatGPT-User) and browser-use agents."
- Training — "a crawler taking your content to train or fine-tune a model," where "your data is permanently absorbed into the underlying architecture" of the AI. GPTBot and ClaudeBot are single-purpose training crawlers.
The September 15 defaults
In July, Cloudflare wrote: "For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default." What shipped on September 15 is a pair of onboarding presets. A new domain that doesn't run ads gets Allow for all three behaviours. A new domain that runs ads gets Search set to Allow, Training set to the new Disallow AI Training, and Agent set to Block on pages with ads. Both presets turn on Bot Preference Sync, and you can change any of them during onboarding or later.
TechCrunch's July report said the new defaults would "apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers." Cloudflare's own September 15 post describes something narrower for existing zones: "Your current settings carry over on their own." Zones that never configured the AI controls and never turned on the legacy block were mapped to Allow for Search, Training and Agent, and earlier Training choices of Block were migrated to Disallow AI Training. Even so, open your own zone's settings and check rather than assume.
Mixed-use crawlers are the trap
In July, Cloudflare warned that "multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors," and that "Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training."
That is now how it works. Since September 15, "Block" and "Block on pages with ads" for Training also apply to mixed-use crawlers, so either one can remove you from Google, Bing and Apple search. The fix is the new Disallow AI Training setting. It publishes a no-training preference in your robots.txt, keeps Applebot, Bingbot and Googlebot crawling for search, and blocks other training crawlers, including the training-only crawlers from Amazon, Anthropic, Meta and OpenAI. One gap: Cloudflare says Bing won't read that robots.txt preference until its support launches, targeted for early 2027.
Where to change it
The dashboard path is /:account/:zone/security/settings (Security → Settings → AI bot policies), available to all customers including Free. Search and Agent each offer Allow, Block on pages with ads, or Block. Training adds a fourth option, Disallow AI Training.
The old toggle is being retired
Cloudflare's docs mark the "Block AI bots" setting as deprecating on September 15, 2026, and Cloudflare's September post says it "will be deprecated in favor of the more granular Search, Training, and Agent controls." Notably, the old toggle "excludes mixed-purpose bots that are used both for Training and for Search," so sites relying on it may have been blocking less than they thought.
Managed robots.txt prepends, it does not replace
Managed robots.txt is now generated by Bot Preference Sync from your Search, Agent and Training policies. It is available on all plans, and if you already serve a robots.txt, Cloudflare "will prepend our managed robots.txt before your existing robots.txt, combining both into a single response." The managed block disallows Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot, and meta-externalagent, and emits Content-Signal: search=yes, ai-train=no, use=reference.
Note that this managed list is training-only. It does not contain OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User, or Perplexity-User. A site owner who enabled managed robots.txt has opted out of training but kept citation eligibility—which is a reasonable default and worth recognizing as such, rather than framing it as a mistake.
Cloudflare is clear about the limits of robots.txt: "robots.txt compliance is voluntary… Some crawler operators may disregard your robots.txt directives and crawl your content regardless." In August 2025, Cloudflare found Perplexity "repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity," running an undeclared stealth crawler at 3–6 million daily requests on IPs outside its published ranges. Cloudflare delisted Perplexity as a verified bot and shipped detection signatures to all customers including Free.
Bot Fight Mode: The Free-Plan Landmine
If you're on Cloudflare's Free plan, there is one setting that overrides every other rule you write: Bot Fight Mode. Cloudflare's documentation states plainly: "You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules."
Bot Fight Mode operates in a separate evaluation pipeline where Skip, Bypass, and Allow actions have no effect. It applies to the entire domain with no endpoint restrictions, and it issues "computationally expensive challenges that force the requesting client to perform CPU-intensive calculations."
This is confirmed independently in the WAF docs: skip rules can bypass rate limiting rules, Managed Rules, and Super Bot Fight Mode rules, but "you cannot skip Bot Fight Mode (available on the Free plan)."
Cloudflare also warns Bot Fight Mode "may challenge API or mobile app traffic," and points AI bot handling to its separate AI bot settings. In other words, Bot Fight Mode does not natively distinguish an AI search crawler from a scraper.
AI crawlers cannot solve JavaScript challenges. The bots that read your pages for ChatGPT, Claude, and Perplexity don't run JavaScript. They get handed the challenge and nothing else. If Bot Fight Mode is on, your permissive robots.txt is irrelevant—the crawler never reaches your origin to read it.
Widely shared walkthroughs recommend creating a Security → WAF → Custom rule with a user-agent expression and a Skip (or Allow) action that is meant to cover WAF managed rules, Bot Fight Mode and AI bot blocking, then checking it with curl. On the Free plan, that contradicts Cloudflare's own documentation. The correct move for a Free-plan reader is to turn Bot Fight Mode off (or upgrade to Super Bot Fight Mode, which is skippable) rather than try to write around it.
The Five-Layer Audit
This is the practical checklist to run on your own site. Ordered from cheapest to most involved, expect this to take 20–30 minutes on a first pass.
Layer 1: Policy (robots.txt)
Fetch your own /robots.txt in a browser and read the whole response, not the file in your repo. Cloudflare prepends its managed block if enabled, so the served file may differ from the committed one.
Check that OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, Googlebot, Bingbot, and Applebot are not caught by a Disallow: /—including via a User-agent: * group.
The classic error: a Disallow under one user-agent group does not apply to others, but a blanket User-agent: * followed by Disallow: / blocks everything unless explicitly overridden. If you've added specific Allow rules for AI search crawlers, they must come after any wildcard disallows in their own user-agent groups.
Layer 2: Edge / CDN
In Cloudflare: Security → Settings holds the Search/Agent/Training policies (all plans, since July 1, 2026), the deprecated "Block AI bots" option and the Bot Fight Mode toggle (filter by bot traffic). If Training is set to Block or Block on pages with ads, switch it to Disallow AI Training unless you really mean to drop out of Google and Bing.
On Free, if Bot Fight Mode is on, your only options are off or Super Bot Fight Mode. No custom rule can skip it.
On Vercel: Both relevant rulesets are off by default. Bot protection managed ruleset is labeled Off, and AI bots managed ruleset is labeled Allow when inactive. Confirm both are still at their defaults unless you've deliberately changed them. If you have switched Vercel's bot protection on, it "identifies clients that violate browser-like behavior and serves a javascript challenge to them"—which is exactly what a non-JS AI crawler fails. Vercel does automatically exclude verified bots from that ruleset.
Stacking Cloudflare in front of Vercel degrades bot detection: "Bot Protection doesn't work when a reverse proxy (e.g. Cloudflare, Azure, or other CDNs) is placed in front of your Vercel deployment," causing obscured detection signals and frequent re-challenges. This is a very common indie-dev topology—if you're running both, your edge behavior is governed by Cloudflare's rules, not Vercel's.
Layer 3: Rate limits
A July 2026 practitioner audit of an AWS WAF site shows blocks that only appear on the 4th or 5th request in a burst. The site served the full 136KB page for the first three requests, then returned "a stub with a JavaScript challenge and an HTTP 202" on requests 4 and 5. Rate limits, not robots.txt, were the block.
A single curl test will pass; a crawl will not. If you're testing, send a short burst of five requests over ten seconds, not one request.
Layer 4: Evidence, not guesses
Cloudflare AI Crawl Control (available on all plans) → Crawlers shows which AI services actually reached your zone. The Directives tab shows robots.txt fetch health (successes vs 400+ responses) and a violations table with bot name, operator, path, and request counts, though "violations are not logged in real-time."
This is the closest thing to ground truth a free-plan user has, because origin logs won't show edge-terminated requests. If OAI-SearchBot appears in your violations table with disallowed paths and non-zero request counts, you have a real block to fix. If it doesn't appear at all, the block may be happening before the crawler identifies itself, or your content simply hasn't been indexed yet.
Layer 5: Rendering
curl your post URL and read the raw HTML. If your post body only appears after JavaScript runs, no major AI crawler except Google's will ever see it.
Vercel and MERJ's analysis covering OpenAI, Anthropic, Meta, ByteDance, and Perplexity bots found that "none of the major AI crawlers currently render JavaScript"—although they do fetch JS files without executing them. Google is the exception, because Gemini surfaces ride on Googlebot's rendering infrastructure. ChatGPT spent 11.50% of requests fetching JS, Claude 23.84%, but neither executes it.
For a static-site blog built with Astro, Next.js static export, or Hugo, this is usually fine—your HTML is already in the initial response. For a SaaS marketing site built as a single-page app where content renders client-side, it usually isn't.
Common mistakes worth naming
- Treating "block AI" as one decision instead of three (training / search / agent).
- Blocking Google-Extended believing it removes you from AI Overviews. It doesn't; it governs Gemini training and grounding. The AI Overviews opt-out is the separate Search Console control.
- Pasting a full ai.robots.txt blocklist (the open-source maintained list) without stripping the retrieval bots. The list is useful as a starting point, but copying it wholesale blocks OAI-SearchBot, PerplexityBot, and Claude-SearchBot along with the training crawlers.
- Trusting a spoofed-user-agent
curlas proof, when the edge verifies by IP, reverse DNS, or cryptographic signature. Vercel documents three bot verification methods, including Web Bot Authentication through RFC 9421 HTTP Message Signatures—meaning acurl -A "GPTBot"test tells you about user-agent-string rules only, not IP-range allowlists or signature validation. - Writing an llms.txt instead of fixing access. Google's AI optimization guide states outright: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search," and "Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." In June 2026, Google's John Mueller called llms.txt "purely speculative for now." He added that the most basic agentic optimization is not blocking agents: "I think that hurdle will be the biggest, for most sites."
The Honest Counter-Case: Should You Block After All?
The data on whether to allow AI crawlers at all is mixed, and the best evidence points in opposite directions depending on your site's profile.
The crawl-to-refer imbalance is real
Cloudflare measured crawl requests against referrals for the first week of August 2025, all industries: Anthropic nearly 50,000:1; OpenAI 887:1; Perplexity 118:1. For News & Publications specifically: Anthropic 2,500:1; OpenAI 152:1; Perplexity 32.7:1.
In that August 2025 analysis, training traffic was "responsible for nearly 80% of the crawling from AI bots." Cloudflare's July 2026 report found that "52% of crawler requests are now for AI training as of June 2026, up from 22% in Spring 2025." Cloudflare has also said that more than half of AI crawler traffic goes to re-fetching pages that haven't changed.
This is a bandwidth and fairness problem. Cloudflare has said more than a million customers turned on its one-click AI crawler block between September 2024 and July 2025.
But blocked sites still get cited—if they're already famous
BuzzStream research from spring 2026 (Citation Labs' XOFU tool, 4 million citations from 3,600 prompts across 10 industries, focused on the top 50 news sites) found:
- 88.2% of sites blocking GPTBot still appeared in citations
- 82.4% of sites blocking OAI-SearchBot still appeared
- 92.3% of sites blocking Google-Extended still appeared
- "Roughly 70% of the citations in our dataset from ChatGPT were from sites that block ChatGPT's retrieval bots," and 95% came from sites that block training bots
- CNBC (blocking multiple bots) appeared 1,298 times; Yahoo (blocking Google-Extended) appeared close to 30,000 times
This study is real and carefully reported. It also does not transfer to a solo founder's SaaS blog. The top 50 news sites are cited because they are quoted, aggregated, syndicated, and summarized everywhere else on the web—the model does not need to fetch nytimes.com to know what the NYT reported. A 40-post indie blog has none of that secondary corpus.
The cloro.dev study (1,058 domains, published July 6, 2026) points the other way and is closer to the audience's situation: median citation propensity (ChatGPT citations per Google-organic appearance, two-day window) was 0.003 for GPTBot-blocking domains vs 0.417 for non-blockers. The authors caveat that the figures are "directional: the corpus reflects cloro's monitored prompt mix rather than the whole web," and that the data can't prove causation. But directionally: if you are not the New York Times, a blanket block on a provider's crawlers likely costs you citations in that provider's answers.
73% of sites have technical barriers blocking AI crawler access
Otterly.ai's AI Citations Report 2026 (January–February 2026, 1m+ AI citations across ChatGPT, Perplexity, and Google AI Overviews) found that 73% of sites have technical barriers blocking AI crawler access. Named barriers: robots.txt blocks, CDN rules blocking non-browser user agents, and JavaScript rendering.
For a small site, the practical risk is a network-layer default or a stale setting enforcing a policy the owner never meant to choose.
What the Numbers Say It's Worth
SE Ranking's 2026 traffic study puts AI platforms at 0.32% of all website traffic in 2026, up from 0.24% in 2025, against 42.75% for organic search. By volume, AI referral traffic is still a rounding error, though SE Ranking measured a 16x rise from 2024 to 2026.
But the intent density is different. Ahrefs' own analytics (30 days before its June 2025 write-up) found that AI search was 0.5% of traffic but 12.1% of signups, a 23× higher conversion rate than traditional organic search.
Ahrefs includes its own caveat, which belongs in any post that cites the 23× figure: "users from AI search click links 75% less than they do in traditional organic search," and the author questions whether the conversion advantage would hold at higher volume.
The short version: low volume, high intent, near-zero cost to stay eligible. The configuration you set today takes an hour. Staying accidentally blocked costs you compounding citation equity you'll never see in your analytics because the request was terminated at the edge.
The Recommended Default for a Small Blog
Allow Search and Agent. Refuse Training with a disallow rather than a hard block, so mixed-use search crawlers keep working. Don't write an llms.txt. Do check Search Console's generative-AI performance reports, which Google rolled out to all websites on August 31, 2026.
Here's a paste-able robots.txt block that allows search and user-fetch crawlers while disallowing training:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: cohere-ai
Disallow: /
Then go check the edge. If you're on Cloudflare:
- Security → Settings: confirm Search and Agent are set to Allow, and Training is set to Disallow AI Training (not Block) if you want to stay in Google and Bing.
- Security → Settings, bot traffic filter: confirm Bot Fight Mode is off, or upgrade to Super Bot Fight Mode and write the skip rule.
- AI Crawl Control → Crawlers: within a week, check which bots are actually reaching your zone.
If you're on Vercel, confirm both managed rulesets are still at their defaults (Off / Allow) unless you've deliberately tuned them.
If you enabled Cloudflare's managed robots.txt, you've already opted out of training and stayed eligible for citations. It now runs through Bot Preference Sync, so read your served /robots.txt to confirm the prepended block is there.
Recheck Your Settings Now
Cloudflare's new defaults took effect on September 15, 2026, and the meaning of "Block" for training changed on the same day. If you're on Cloudflare and haven't looked at Security → Settings since then, do it now. That goes double if your blog earns revenue from ads that Cloudflare may detect as "pages that display ads." Confirm the settings applied to your zone match your policy, and recheck whenever Cloudflare announces a change to its AI controls.
The question is no longer "should I block AI crawlers?" The question is: have you set a policy at every layer that can enforce one, or are you inheriting defaults chosen by your CDN, your host, and your firewall provider?
Your robots.txt is not the file that decides this. Your edge is, and its answer changed on September 15.
Some small sites publish via GitHub pull requests against a static-site repo, where robots.txt lives in version control and the publish button is a merge. For them there's a parallel discipline. A permissive robots.txt does nothing if the edge blocks the crawler, and a well-cited post does nothing if the links inside it 404 when a reader (or an AI) follows them.
Magic Share's link verification runs the same kind of audit one layer up: every cited URL in a draft is resolved before the post is saved, and a verification report ships attached. Posts that 403 for crawlers and citations that 404 for readers fail the same way: invisibly, and with costs that compound over time. Vercel's data showing ChatGPT spending 34.82% of fetches on 404 pages makes the link-hygiene point concrete.
If you're running a blog that compounds when it's tended year-round rather than in bursts, AI citation eligibility is a standing configuration, not a one-time task. Google now measures generative-AI impressions separately in Search Console. The posts you publish this month may be cited six months from now by a user asking a question you didn't anticipate, through a retrieval bot you forgot you allowed. The harvest is delayed, but the decision to leave the gate open is made today.
This post is also part of a broader question: whether AI-assisted content is allowed at all under Google's scaled content abuse policy. That post covered whether Google penalizes AI-drafted posts; this one covers whether AI crawlers can reach them. Together they form the two gates: policy and access. Both have to be open for a post to compound.
FAQ
If I block GPTBot but allow OAI-SearchBot, will my content still be used for training?
Possibly, through indirect pathways. Blocking GPTBot prevents OpenAI's direct crawling for training while OAI-SearchBot keeps you in ChatGPT search. But if your content is cited in other indexed sources, quoted in public forums, or included in datasets assembled by third parties, it may still reach training corpora indirectly. The block is real but not hermetic.
Does blocking training crawlers hurt my rankings in regular Google Search?
No. Google states explicitly that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Blocking training-only crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) has no direct effect on your visibility in traditional search results or in AI Overviews, which are governed by Googlebot's indexing and your Search Console generative-AI toggle. One exception: on Cloudflare, a Training setting of Block now also stops Googlebot, so use Disallow AI Training instead.
I use Cloudflare's Free plan and Bot Fight Mode is on. Can I write a custom rule to allow AI search crawlers?
No. Cloudflare's documentation says you "cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules," so your options are to turn it off or upgrade to the paid Super Bot Fight Mode, which custom rules can skip.
My robots.txt allows all crawlers, but I don't see AI traffic in my logs. What's blocking them?
Usually an edge challenge (Bot Fight Mode, Vercel bot protection or WAF rate limits) that stops the request before it reaches your origin, client-side rendering that non-JavaScript crawlers can't read, or a new site that crawlers haven't discovered yet. The five-layer audit above checks each of these in order.
Does the September 15 Cloudflare change affect my site if I don't run ads?
Probably not. For an existing zone, Cloudflare says your current settings carried over, and a zone that never touched the AI controls was mapped to Allow. A new domain without ads is offered Allow for all three behaviours, while one that runs ads is offered a preset that disallows training and blocks agents on pages Cloudflare detects as serving ads. Either way, check Security → Settings. A Training setting of Block or Block on pages with ads now also stops Googlebot, Bingbot and Applebot.