- Home
- Robots.txt Checker
Robots.txt Checker
Test any page against its site's robots.txt for twelve crawlers at once: Google and Bing, and the AI crawlers from OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl and ByteDance. For each one you see allowed or blocked and the exact rule that decided it.
How the check works
We fetch /robots.txt from the site of the URL you enter, following redirects the way crawlers do, and parse it according to RFC 9309, the robots exclusion standard that Google and Bing follow. For each crawler we find the user-agent group that applies to it, then the most specific Allow or Disallow rule that matches the path you entered, including * wildcards and the $ end anchor. The table shows the result and the rule, and the full file is printed below it so you can see the context.
The crawlers we test
| User agent | Who | What it is for |
|---|---|---|
| Googlebot | Google Search | |
| Bingbot | Microsoft | Bing, and the search behind several AI assistants |
| Google-Extended | Opt-out token for Gemini training; does not affect Search | |
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | Pages fetched when a user asks ChatGPT to open them |
| ClaudeBot | Anthropic | Model training |
| Claude-SearchBot | Anthropic | Claude search results |
| PerplexityBot | Perplexity | Perplexity search index |
| CCBot | Common Crawl | Open web archive used by many AI models |
| Applebot-Extended | Apple | Opt-out token for Apple AI training |
| Bytespider | ByteDance | Crawling for ByteDance products and models |
Mistakes we see most often
- A staging rule left in production.
Disallow: /underUser-agent: *blocks every crawler from the whole site. It is the most damaging one-line mistake in SEO. - Blocking CSS and JavaScript. Google renders pages, and blocked assets can make a page look broken to it.
- Expecting robots.txt to hide pages. Blocked pages can still be indexed from links. Use noindex for that.
- A specific group that forgets the basics. Once you add a group for Googlebot, Googlebot ignores the * group, so any rule you want it to follow must be repeated.
- No Sitemap line. Not an error, but listing your sitemap in robots.txt is the easiest way to help every crawler find your pages.
Search engines versus AI crawlers
Since 2023 most AI companies have published separate user agents for training and for search. That lets you make a choice instead of an all-or-nothing decision: you can keep your articles out of training data while still being fetched, quoted and linked when someone asks an AI assistant a question your page answers. If AI answers are becoming a source of visitors for you, test the search agents, not just the training ones.
Related tools
Check a live site
Redirect Checker
Every hop of a redirect chain with status codes, loops and HTTPS problems.
Open Graph Checkercoming soon
Preview how a link looks when shared.
Frequently asked questions
Does blocking a page in robots.txt remove it from Google?
No. robots.txt stops crawling, not indexing. A blocked page can still appear in results, without a description, if other pages link to it. To keep a page out of the index, let it be crawled and add a noindex robots meta tag or header.
Should I block AI crawlers?
It depends on what you want. Blocking training crawlers such as GPTBot, ClaudeBot, Google-Extended or CCBot keeps your content out of future training sets. Blocking search and browsing agents such as OAI-SearchBot, Claude-SearchBot or PerplexityBot can also remove you from AI answers that would otherwise cite and link to you. Many sites block training and allow search.
Do all crawlers obey robots.txt?
Reputable search engines and the major AI companies say they do. Scrapers and malicious bots often do not. robots.txt is a request, not access control; protect private content with authentication.
What happens if robots.txt is missing?
A 404 means “no rules”, so everything may be crawled. A 5xx server error is treated more cautiously: Google pauses crawling of the whole site until robots.txt answers again.
Why does a more specific rule win?
Google and the RFC 9309 standard pick the rule with the longest matching path. If two rules are equally long, Allow wins. A crawler that has its own User-agent group ignores the * group completely.