No Upload, No Wait
Everything runs on your own device, so there is no queue and no server round-trip.
- 100% FreeNo hidden charges
- Stays on Your DeviceNothing is uploaded
- Easy to UseSimple and quick
- Any DeviceBrowser, mobile and tablet
Paste a robots.txt, give it a crawler and a URL path, and this tells you whether the crawler is allowed — and, more usefully, which single rule decided it. Rule precedence is the part people get wrong, and most testers never show their working.
Path only, starting with a slash — not the full address.
Nothing is uploaded. Your robots.txt is parsed in your browser and never sent anywhere.
The rule that wins is not the first one
This is the single most common misunderstanding about robots.txt, and it causes real damage in both directions — pages blocked that should not be, and pages crawled that were meant to be private.
Three things decide the outcome, in this order:
1. Only one group applies
A crawler obeys exactly one group of rules: the one whose User-agent line matches it most specifically. If your file has a block for * and a block for Googlebot, then Googlebot follows only the Googlebot block and ignores the wildcard block completely — including any Disallow lines you assumed applied to everyone.
This trips people constantly. Adding a small Googlebot group to set a crawl delay silently exempts Googlebot from every rule in your wildcard group.
2. The longest matching rule wins
Within that group, order on the page is irrelevant. The rule with the longest path pattern wins. That is why this works:
| Rule | Length | Outcome for /wp-admin/admin-ajax.php |
|---|---|---|
Disallow: /wp-admin/ | 11 | Matches, but shorter |
Allow: /wp-admin/admin-ajax.php | 27 | Wins — crawling allowed |
3. On an exact tie, Allow wins
If two rules match with the same length and disagree, the crawler is permitted.
Wildcards, and the one that is easy to misread
* matches any run of characters. $ anchors the rule to the end of the path.
The trap is that Disallow: /*.pdf$ blocks /report.pdf but not /report.pdf?v=2, because the query string means the path no longer ends at .pdf. If you are trying to keep PDFs out of search, that is a gap worth knowing about.
A rule without a trailing slash is also a prefix, not a folder. Disallow: /search blocks /searchable-page too.
What robots.txt does not do
It is not a privacy control and it does not remove pages from Google. A blocked URL can still appear in results if other sites link to it — Google just cannot read the page to describe it. To keep something out of the index, allow crawling and use a noindex tag; to keep something private, put it behind a login. Blocking a page in robots.txt actually prevents Google from seeing a noindex tag on it.
Frequently asked questions
Is my robots.txt sent to your server?
No. The parsing happens entirely in your browser. You can disconnect from the internet once the page has loaded and it still works.
Why does my Googlebot rule seem to ignore my wildcard rules?
Because it does. A crawler obeys only the most specific group that matches it. Once a Googlebot group exists, Googlebot stops reading the group for * entirely. Any rule you want to apply to Googlebot must be repeated inside its own group.
Does the order of lines matter?
No. Within a group, the longest matching path wins regardless of where it appears. Ties go to Allow.
What does an empty Disallow mean?
Disallow: with nothing after it places no restriction at all, which is the same as allowing everything for that group.
Will blocking a page remove it from Google?
No. It stops Google reading the page, but the URL can still be listed if other pages link to it. Use a noindex tag on a crawlable page to remove it from the index.
Does this follow Google’s rules exactly?
It implements the behaviour described in Google’s robots.txt specification: most-specific group, longest match, ties to Allow, plus * and $ wildcards. Other crawlers vary, and some ignore robots.txt entirely.
Related tools
- Robots.txt Generator — build the file in the first place
- AI Crawler Checker — which AI bots your file allows
- XML Sitemap Generator
- Amazon Affiliate Link Checker
- Every tool here that never uploads your data
Why use our Robots.txt Tester?
Genuinely Private
What you enter never leaves your browser. We could not see it even if we wanted to.
No Registration
No account, no email, no usage limit. Open it and use it.
Works Everywhere
Any modern browser on desktop, tablet or phone. Nothing to install.