No Upload, No Wait
Everything runs on your own device, so there is no queue and no server round-trip.
- 100% FreeNo hidden charges
- Stays on Your DeviceNothing is uploaded
- Easy to UseSimple and quick
- Any DeviceBrowser, mobile and tablet
Paste your robots.txt below and this checks it against the 14 AI crawlers that matter in 2026 — telling you which you allow, which you block, and whether you have accidentally blocked the ones that actually send you traffic.
That last part is the reason this tool exists. Training crawlers and search crawlers look nearly identical in a robots.txt file, and blocking the wrong one is the quietest way to disappear from AI search results.
Find yours at yoursite.com/robots.txt. Select all, copy, paste above.
This runs entirely in your browser. Nothing you paste is uploaded or stored — here is how to verify that.
Search & citation crawlers
These send traffic back. They index your pages so AI assistants can cite and link you. Blocking one of these makes you invisible in that assistant’s answers.
Training crawlers
These collect content to train models. They return little or no direct traffic. Blocking them is a legitimate choice and does not affect whether AI search engines can cite you.
Training crawlers and search crawlers are not the same thing
This is the distinction the whole thing hinges on, and it is worth thirty seconds.
Training crawlers — GPTBot, ClaudeBot, Google-Extended, CCBot and others — collect content to train models. They send you almost nothing in return. Blocking them is a legitimate decision that a lot of publishers make.
Search and citation crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot and their user-triggered counterparts — index your pages so an AI assistant can cite you and link back. These behave much more like Googlebot: they take content and return visitors.
The expensive mistake. Someone decides to opt out of AI training, adds a blanket block, and unknowingly removes their site from ChatGPT, Claude and Perplexity answers as well. The traffic loss is invisible because nothing breaks — you simply stop being cited. This tool exists to catch exactly that.
The 14 crawlers this checks
| Crawler | Operator | What it does |
|---|---|---|
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results |
| ChatGPT-User | OpenAI | Fetches a page when a user opens your link |
| GPTBot | OpenAI | Model training |
| Claude-SearchBot | Anthropic | Indexes pages so Claude can cite you |
| Claude-User | Anthropic | Fetches a page for a Claude user |
| ClaudeBot | Anthropic | Model training |
| PerplexityBot | Perplexity | Indexes pages for citation |
| Perplexity-User | Perplexity | Fetches a page for a user |
| Google-Extended | Gemini training. Does not affect Google Search | |
| Applebot-Extended | Apple | Apple model training |
| Bytespider | ByteDance | Training, reported as high-volume |
| Meta-ExternalAgent | Meta | Model training |
| CCBot | Common Crawl | Open dataset many models train on |
| Amazonbot | Amazon | Alexa and model work |
How to read the result
You are blocking search crawlers
Almost always an accident. Remove those rules unless you have a specific reason to be absent from AI search. Blocking a search crawler does not protect your content from training — it just stops you being cited.
You are blocking training crawlers only
This is the configuration most publishers want. You have opted out of model training while staying visible and citable in AI search.
Nothing is blocked
A valid choice. Maximum visibility everywhere, and your content is available for training too. Plenty of sites decide the citations are worth it.
One honest limitation.
robots.txt is a request, not a wall. Well-behaved crawlers from OpenAI, Anthropic, Google and Perplexity respect it. Anything that ignores robots.txt will ignore it here too — blocking that requires server-level or firewall rules, not a text file.Making changes safely
Three rules that prevent most problems:
Name the specific crawler. Put rules under the exact user-agent token rather than a blanket User-agent: * block, so you are not catching Googlebot in the same net.
Check after every change. A single misplaced line can invert your intent. Run your file back through this tool once it is live.
Do not confuse Google-Extended with Googlebot. Google-Extended controls Gemini training only. It has no effect on Google Search ranking.
If you need to build a file from scratch, our robots.txt generator will produce one, and the guide to generative engine optimisation covers the wider strategy of being cited by AI search.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot, which is a separate user-agent. You can block one and allow the other, and OpenAI documents this deliberately.
Does Google-Extended affect my Google rankings?
No. Google-Extended controls whether your content is used for Gemini model training. Google Search crawling and ranking are governed by Googlebot, which is unaffected.
Is my robots.txt uploaded anywhere?
No. The check runs entirely in your browser using JavaScript on this page. Nothing you paste is sent to a server, and you can confirm that yourself — open your browser’s Network tab, run the check, and watch for an outbound request. There will not be one.
Should I block AI training crawlers?
There is no universally right answer. Blocking protects your content from being used as training data but returns nothing. Allowing it costs you control but may improve how well models understand your site. The decision that is rarely right is blocking the search crawlers, since those send traffic back.
Where do I find my robots.txt?
Add /robots.txt to your domain — for example yoursite.com/robots.txt. If nothing loads, you do not have one, which means every crawler is allowed by default.
Do these rules actually get obeyed?
By the major operators, yes — OpenAI, Anthropic, Google and Perplexity all publish their tokens and honour them. Crawlers that ignore robots.txt need blocking at the server or firewall level instead.
Related tools and guides
- Robots.txt Generator — build a file from scratch
- Generative Engine Optimisation guide — the strategy behind this
- XML Sitemap Generator
- Meta Tag Generator
- Tools that never upload your files
The Robots.txt Tester checks one crawler against one URL and shows which rule won.
Why use our AI Crawler Checker?
Genuinely Private
What you enter never leaves your browser. We could not see it even if we wanted to.
No Registration
No account, no email, no usage limit. Open it and use it.
Works Everywhere
Any modern browser on desktop, tablet or phone. Nothing to install.