The HN Solutions Robots.txt Generator builds a correct, valid robots.txt file in seconds. Start from a ready-made preset — allow everything, a WordPress-optimised setup, block AI training bots like GPTBot, or block all crawlers — then add your own disallowed paths and your sitemap URL, and copy or download the finished file.
Why our robots.txt generator is better
- One-click presets — including a WordPress preset and an up-to-date “block AI bots” list (GPTBot, ClaudeBot, CCBot, Bytespider and more).
- Correct syntax guaranteed — no typos or invalid directives that could accidentally block your whole site.
- Sitemap line included — so search engines find every page faster.
- Copy or download a ready-to-upload robots.txt file.
- Free and instant, generated fully in your browser.
How to create a robots.txt file
- Pick a preset that matches your site.
- Add your sitemap URL and any extra paths to block.
- Copy or download the file and upload it to your site’s root folder.
Frequently asked questions
What is a robots.txt file?
Robots.txt is a plain text file at the root of your website that tells search engine crawlers which parts of the site they may or may not crawl.
Where do I put robots.txt?
In the root directory of your domain, so it loads at yoursite.com/robots.txt. On WordPress, many SEO plugins like Rank Math can save it for you.
Can robots.txt block AI bots from using my content?
Yes. The “Block AI bots” preset adds rules for crawlers like GPTBot, ClaudeBot, Google-Extended and CCBot that collect content for AI training.
Does robots.txt remove pages from Google?
Not by itself. It stops crawling, but to keep a page out of search results you should use a noindex meta tag instead.
Is this robots.txt generator free?
Yes, completely free with no sign-up and no limits.
See all our free utilities in the SEO tools hub and daily tools hub.
The mistakes that deindex a site
Disallow: / under User-agent: * blocks your entire site. It is the correct setting on a staging server and catastrophic in production — and it reaches production regularly, because staging files get copied across during a launch. Check yours after any migration or redesign.
Blocking CSS and JavaScript. Google renders pages before judging them. If your theme assets are blocked, it sees a broken layout and rates the page accordingly.
Assuming Disallow means private. It asks crawlers not to fetch a URL. It does not hide it, and a disallowed page can still appear in results if other sites link to it. To keep a page out of the index, use a noindex meta tag — and note that the page must be crawlable for Google to see that tag at all.
Blocking AI training crawlers
Robots.txt has become the standard place to state whether AI companies may use your content for training. The major crawlers respect it by convention rather than obligation:
GPTBot — OpenAI trainingClaudeBot — AnthropicGoogle-Extended — Gemini training, separate from GooglebotCCBot — Common Crawl, which feeds many datasets
Blocking Google-Extended does not affect normal Search ranking. Blocking Googlebot very much does — do not confuse them.
Where the file must live
The domain root, and nowhere else: yoursite.com/robots.txt. A file in a subfolder is ignored entirely. Subdomains need their own — blog.yoursite.com does not inherit from the main domain.
WordPress serves a virtual robots.txt by default. If you upload a physical file it takes over; if you use Rank Math or Yoast, edit through the plugin so you are not maintaining two versions that disagree.
Keep it short
Every rule is a chance to be wrong. A long robots.txt accumulated over years usually contains lines nobody can explain and at least one that blocks something it should not. Most sites need a handful of rules and a sitemap line.
After changing it, verify at yoursite.com/robots.txt and check Search Console’s robots report, which shows the file Google actually fetched and any parse errors. Pair it with the XML sitemap generator to point crawlers at what you do want indexed.