Upload robots.txt to the root of your site so it is reachable at yoursite.com/robots.txt.
The HN Solutions Robots.txt Generator builds a correct, valid robots.txt file in seconds. Start from a ready-made preset — allow everything, a WordPress-optimised setup, block AI training bots like GPTBot, or block all crawlers — then add your own disallowed paths and your sitemap URL, and copy or download the finished file.
Why our robots.txt generator is better
- One-click presets — including a WordPress preset and an up-to-date “block AI bots” list (GPTBot, ClaudeBot, CCBot, Bytespider and more).
- Correct syntax guaranteed — no typos or invalid directives that could accidentally block your whole site.
- Sitemap line included — so search engines find every page faster.
- Copy or download a ready-to-upload robots.txt file.
- Free and instant, generated fully in your browser.
How to create a robots.txt file
- Pick a preset that matches your site.
- Add your sitemap URL and any extra paths to block.
- Copy or download the file and upload it to your site’s root folder.
Frequently asked questions
What is a robots.txt file?
Robots.txt is a plain text file at the root of your website that tells search engine crawlers which parts of the site they may or may not crawl.
Where do I put robots.txt?
In the root directory of your domain, so it loads at yoursite.com/robots.txt. On WordPress, many SEO plugins like Rank Math can save it for you.
Can robots.txt block AI bots from using my content?
Yes. The “Block AI bots” preset adds rules for crawlers like GPTBot, ClaudeBot, Google-Extended and CCBot that collect content for AI training.
Does robots.txt remove pages from Google?
Not by itself. It stops crawling, but to keep a page out of search results you should use a noindex meta tag instead.
Is this robots.txt generator free?
Yes, completely free with no sign-up and no limits.
See all our free utilities in the SEO tools hub and daily tools hub.