Is your site blocking the AI crawlers?
Enter any website. We request it once as each named AI crawler and show you the HTTP status it gets back, next to the robots.txt rule that applies to it. No signup, no email, no card.
Reading robots.txt is not enough on its own: a firewall or CDN rule can refuse a crawler by name without anything appearing in that file, and Cloudflare has blocked AI crawlers by default for new domains since 1 July 2025. This checks both layers, per crawler. How we do this
What each crawler actually controls
The mistake that costs businesses the most is treating "AI crawlers" as one thing. For OpenAI, Perplexity and Anthropic, the crawler that decides whether you can be cited in an answer is a different user agent from the one that collects training data. Blocking the training crawler is a reasonable choice about your content. Blocking the retrieval crawler removes you from the answers your customers are reading.
- OAI-SearchBot OpenAI · retrieval
Decides whether your site can be cited in ChatGPT search answers. Blocking this removes you from ChatGPT search. - PerplexityBot Perplexity · retrieval
Perplexity’s indexing crawler. Perplexity states it is not used for model training — blocking it only costs you citations. - Claude-SearchBot Anthropic · retrieval
Indexes pages for Claude’s web search. Blocking it removes you from Claude’s search results. - ChatGPT-User OpenAI · user-triggered
Fetches your page when a user or a ChatGPT action asks for that specific URL. - GPTBot OpenAI · training
Training data only. Blocking GPTBot is a legitimate opt-out and does NOT remove you from ChatGPT search. - ClaudeBot Anthropic · training
Training data only. Blocking it is an opt-out, not a search removal. - Googlebot Google · retrieval
Feeds Google Search AND its AI Overviews and AI Mode — there is no separate Gemini crawler. Blocking it removes you from both. - Bingbot Microsoft · retrieval
Bing’s index, which has historically fed ChatGPT’s web search and Copilot.
Crawler names, roles and behaviour are taken from each operator's own documentation: OpenAI, Perplexity, Anthropic and Google. Last reviewed 2026-09-14.
The four ways this goes wrong
- A leftover blanket block. User-agent: * / Disallow: / from a staging site, or from a "block AI scrapers" plugin someone installed. It stops all three citation crawlers at once.
- Blocking GPTBot and thinking that is the AI opt-out. GPTBot is training only. OpenAI is explicit that you can allow OAI-SearchBot to appear in search results while disallowing GPTBot.
- A CDN or firewall default. Cloudflare has blocked AI crawlers by default for new domains since 1 July 2025. It is invisible in robots.txt, and a business whose web person moved the site behind a CDN may have no idea.
- Bot-management rules that challenge or 403 crawlers as a side effect of stopping scrapers.
If you want the longer version, including what to do about each of these, that is in the technical guide. If you want to know which competitors AI names instead of you, that is the report. If you want to see when your site last changed, that is the last-updated check. If you are comparing tools, we keep an honest roundup of the free ones and comparisons with the paid platforms.
Frequently asked questions
How do I check if my website is blocking AI crawlers?
Enter your address above. This tool requests your homepage once as each named AI crawler — OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, Googlebot and Bingbot — and shows the HTTP status each one received, next to the robots.txt directive that applies to it. Reading robots.txt alone is not enough, because a firewall or CDN rule can refuse a crawler without anything appearing in that file.
Is this tool really free?
Yes. No signup, no email, no card, and the whole result is on screen. We sell a paid report that answers a different question — whether AI engines name your business for real buyer searches in your city, and who they name instead — and this tool exists partly so you can see we measure things honestly before you consider buying anything.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects training data. OAI-SearchBot is the crawler behind ChatGPT search results. Blocking GPTBot is a legitimate choice to keep your content out of model training, and it does not remove you from ChatGPT search. Blocking OAI-SearchBot does remove you. Many sites block one believing they blocked the other, which is the most common mistake this tool surfaces.
Does blocking AI crawlers affect my Google ranking?
Blocking Googlebot does, severely — it also removes you from AI Overviews and AI Mode, because there is no separate Gemini crawler. The robots.txt token Google-Extended is different: it controls Gemini and Vertex grounding and training, and Google states it does not affect Google Search. Blocking OpenAI, Perplexity or Anthropic crawlers has no effect on Google at all.
Why does my robots.txt look fine but a crawler still shows as blocked?
Because the block is in front of your site rather than in the file. Cloudflare has blocked AI crawlers by default for new domains since 1 July 2025, and firewall or bot-management rules can refuse a crawler by user agent. Neither appears in robots.txt. That is why this tool reports an actual HTTP status per crawler as well as the directive.
My results say "could not tell" — what does that mean?
It means we refused to guess. If your site returned an error or a refusal to an ordinary browser request too, then a refusal to a crawler proves nothing specific, so we say so instead of reporting a block. A 502, 503 or 504 also means unavailable rather than refused. Try again in a few minutes.
Does this tool tell me whether ChatGPT recommends my business?
No, and nothing free does that properly. This tool answers whether the engines can read you at all, which is the precondition. Whether they actually name you when a customer asks "who is the best plumber in Tulsa" takes running those searches across the engines several times, which is what our paid report does.