A robots.txt file sits at the root of a website (for example example.com/robots.txt) and tells crawlers which parts of the site they may visit. It’s one of the first files Google requests. A good one points crawlers to your sitemap and keeps them out of pages that shouldn’t appear in search, such as admin areas, carts and internal search results.
This robots.txt generator builds a correctly formatted file from a few choices, warns you about common mistakes, and can add rules to block AI crawlers.
How to create a robots.txt file
- Leave “Allow crawling” selected for a normal public website.
- Add your sitemap’s full URL.
- List any folders or pages crawlers should skip, one per line, starting with a slash.
- Choose whether to block AI crawlers.
- Download the file and upload it to the root folder of your site, so it opens at
yourdomain.com/robots.txt.
How robots.txt rules work
- User-agent names the crawler a group of rules applies to.
*means every crawler. - Disallow blocks a path and everything under it.
Disallow: /admin/blocks the admin folder;Disallow: /blocks the whole site; an emptyDisallow:allows everything. - Allow makes an exception inside a blocked folder.
- Sitemap gives the full address of your XML sitemap.
- A crawler follows the most specific group that matches its name, so a bot with its own group ignores the
*rules.
Robots.txt doesn’t hide pages
Blocking a page in robots.txt stops crawling, not indexing. If other sites link to a blocked page, Google can still list its URL without a description. To keep a page out of search results, let Google crawl it and add a noindex robots meta tag, which you can create with the meta tag generator. And never rely on robots.txt for privacy: the file is public, so it can point curious visitors straight to the folders you list.
Should you block AI crawlers?
AI companies use crawlers to collect training data and to fetch pages for AI search answers. Blocking training crawlers such as GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google’s AI models; it doesn’t affect Google Search) and CCBot (Common Crawl) asks them not to use your content for training. Blocking AI search crawlers like OAI-SearchBot and PerplexityBot can remove your pages from those tools’ answers and the visitors they send. Many publishers block training but allow search. Respectable crawlers obey robots.txt, but it’s a request, not a lock.
After uploading, check the file with the SEO checker, which confirms that robots.txt is found and lists your sitemap.
Frequently asked questions
Where do I upload robots.txt?
To the root of your domain, the same folder as your homepage. Subdomains need their own file. In WordPress, Shopify and similar platforms, use the platform’s robots.txt setting or an SEO plugin instead.
Does Google support Crawl-delay?
No. Google ignores it. Bing and some other crawlers respect it. If Googlebot is overloading your server, contact your host; Google slows down automatically when a server responds slowly or with errors.
Do I need a robots.txt file at all?
No, a site works without one and everything is crawlable by default. But a simple file that lists your sitemap is good practice and avoids 404 errors in your logs.