How to use AI crawler checker
- Enter a website address, such as example.com.
- Select Check AI crawler access.
- Read which crawlers are allowed or blocked, what each one is used for, and the robots.txt rule that decided it.
Three kinds of AI crawler
AI companies now run separate crawlers for separate jobs, and robots.txt can treat them differently. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot collect pages that may be used to train future models. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so an assistant can find and cite them when answering. Assistant fetchers such as ChatGPT-User, Claude-User and Perplexity-User read a page because a person asked the assistant about it.
Blocking training crawlers is a reasonable choice if you don't want your writing used for training. Blocking search and assistant crawlers is different: it removes your site from the answers people get from ChatGPT, Claude and Perplexity. Many sites do both by accident with a blanket rule.
How the check works
Our server fetches the site's public robots.txt and home page, and your browser evaluates the rules for each crawler using RFC 9309, the robots.txt standard: a crawler follows the group that names it exactly, or the * group when none does; the longest matching path rule wins; and Allow wins a tie. The table shows, for each crawler, whether it may read the home page and which rule decided.
The check also reads the home page's X-Robots-Tag header and robots meta tags, and warns if the page asks search engines not to index it. Only the address you type is sent to our server, only public websites can be fetched, and nothing is stored.
What to do with the result
If an assistant you care about is blocked, add a group for its search and user crawlers with Allow: /, or remove the Disallow line that catches it. Changes take effect the next time each crawler reads your robots.txt, usually within a day. Being readable is the first step to being cited; the next is having clear, answer-first pages for the questions your customers ask.
Example
A robots.txt that blocks GPTBot, ClaudeBot and Google-Extended but says User-agent: * Allow: / still lets OAI-SearchBot, Claude-SearchBot and PerplexityBot in, because they aren't named and fall under the * group. A file with a single User-agent: * Disallow: / blocks every crawler, AI or not.
Questions
- What does blocking an AI crawler change?
- Blocking a training crawler (GPTBot, ClaudeBot, Google-Extended) asks that your pages not be used to train models. Blocking a search or assistant crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User) stops those assistants reading your pages to answer questions, so they can't cite you.
- What is sent to your server?
- Only the web address you type. Our server fetches that site's public robots.txt and home page, returns them to your browser, and stores nothing. Only public websites can be checked.
- How are the rules interpreted?
- As RFC 9309, the robots.txt standard: a crawler follows the group that names it, or the * group if none does; the longest matching rule wins, and Allow wins a tie.