robots.txt generator and tester
Build your site's robots.txt (blocking AI robots if you want) and test whether a robot may visit an address, by the official rules.
What is robots.txt?
It is a text file at the site root (example.com/robots.txt) that tells robots, such as Google's, Bing's and AI crawlers, which parts of the site they may visit. Before reading any page, a well-behaved robot reads this file and follows it. Since 2022 the reading rules are official: RFC 9309.
What it does
It organizes the robots' work: keeps the admin panel, the API, the cart and duplicate pages out of search, points to the sitemap and can ask AI robots not to use your content.
What it does not do
It is not a lock: it is a public request, and malicious robots ignore it. It does not remove a page from Google either: if someone links to it, the address may show up in search even when blocked. That is what noindex is for. And never list secret paths here: anyone can read the file.
AI robots: training, search and "on request"
AI companies use different robots for different things. Blocking the training ones does not remove your site from regular search or from the assistants' search. The ones that fetch "on request" (when someone pastes your link into a chat) may ignore robots.txt, according to the companies themselves. The list was checked against each company's documentation (links at the end of the page).
| Robot | Company | What it is for |
|---|---|---|
Googlebot | Regular search | |
Bingbot | Microsoft | Regular search |
Applebot | Apple | Regular search |
GPTBot | OpenAI | AI training |
ClaudeBot | Anthropic | AI training |
Google-Extended | AI training | |
Applebot-Extended | Apple | AI training |
meta-externalagent | Meta | AI training |
CCBot | Common Crawl | AI training |
OAI-SearchBot | OpenAI | AI assistants' search (cites and links the site) |
Claude-SearchBot | Anthropic | AI assistants' search (cites and links the site) |
PerplexityBot | Perplexity | AI assistants' search (cites and links the site) |
meta-webindexer | Meta | AI assistants' search (cites and links the site) |
ChatGPT-User | OpenAI | On user request (may ignore robots.txt) |
Claude-User | Anthropic | On user request (may ignore robots.txt) |
Perplexity-User | Perplexity | On user request (may ignore robots.txt) |
meta-externalfetcher | Meta | On user request (may ignore robots.txt) |
What about sitemap.xml?
It is the site map: an XML file listing the pages you want search engines to know about, with the date of the last change. robots.txt points to it (the Sitemap line). It is not generated here because it needs the real list of your site's pages: the site itself generates it (WordPress, Astro, Next and the like have a plugin or built-in feature) or a program that walks through the site. From the browser it is not possible to walk another site, and this tool sends nothing to any server.
robots.txt
Test
Paste a robots.txt (the one generated above is already here), pick the robot and the address. The answer follows RFC 9309: the robot's group applies (or "*"), and within it the most specific rule.
Things to check
By using the tools, you agree to the terms of use.