How to check if AI crawlers can read your site
Last updated: October 25, 2026
If an AI search engine cannot fetch your pages, it cannot quote them. Most of the time the reason is one line in robots.txt that was added to block "AI" in general, and ended up blocking AI search too. This guide shows what to look for and how to fix it.
Key point: AI companies now run separate crawlers for different jobs. Blocking the crawler used for training does not have to block the one used for search answers. You can choose each one.
The crawlers that matter
Each crawler identifies itself with a user-agent token. These are the ones we check in the free audit. Operators change these lists from time to time, so check their documentation before you make big changes.
| Token in robots.txt | Operator | What it is for |
|---|---|---|
OAI-SearchBot | OpenAI | Showing sites in ChatGPT search answers |
GPTBot | OpenAI | Collecting content that may be used to train models |
ChatGPT-User | OpenAI | Fetching a page when a user asks ChatGPT to; OpenAI says robots.txt rules may not apply |
Google-Extended | A control token, not a separate crawler: governs use for Gemini apps and Vertex AI. Google says it does not affect Google Search | |
ClaudeBot, Claude-SearchBot, Claude-User | Anthropic | Training, search and user-requested fetches |
PerplexityBot, Perplexity-User | Perplexity | Search index and user-requested fetches |
Applebot-Extended | Apple | A control token for use of content in Apple's AI models |
CCBot | Common Crawl | An open web archive that many AI datasets are built from |
meta-externalagent, Bytespider, Amazonbot | Meta, ByteDance, Amazon | Crawlers used for each company's AI products |
Step 1: open your robots.txt
Go to https://yourdomain.com/robots.txt. If you get a 404, nothing is blocked by robots.txt. If your homepage loads instead of a text file, your server is answering every missing file with the homepage, which some crawlers will misread. Ask your developer to return a real 404 or a real robots.txt.
Step 2: look for blanket blocks
These patterns block more than most site owners intend:
User-agent: *
Disallow: /This blocks every crawler that follows robots.txt, including search engines. It is common on staging sites that were copied to production by mistake.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: PerplexityBot
Disallow: /This was often added as an "opt out of AI" rule. It also removes the site from ChatGPT search and Perplexity answers.
Step 3: decide training and search separately
If you want to appear in AI search answers but keep your content out of model training, a starting point looks like this:
# Allow AI search and user-requested fetches
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /Robots.txt uses the most specific group that matches a crawler. A crawler listed in its own group ignores the User-agent: * group, so test the result instead of guessing.
Step 4: check the page itself
Robots.txt is not the only switch. Also check:
- Meta tags such as
noindex, ornoaiandnoimageai, which some sites use to signal an opt-out. - The X-Robots-Tag header, which can carry the same instructions for the whole response.
- Content that needs JavaScript. Many crawlers read the raw HTML. If your main text only appears after scripts run, they may see an empty page.
- Firewall and bot protection rules at your host or CDN, which can block crawlers before robots.txt is read.
What this does and does not do
Letting crawlers in is housekeeping. It makes it possible for AI engines to read and cite you. It does not make them recommend you. Being mentioned still depends on whether other sites, reviews and articles talk about you.
Sources
- OpenAI: Overview of OpenAI crawlers
- Search Engine Roundtable: Google-Extended does not affect Google Search
Check your own site in one click. The free AI Seen Score extension runs these checks on the page you are viewing: AI crawler rules in robots.txt, llms.txt, structured data, titles and headings, and content that only appears after JavaScript runs. It runs in your browser and sends nothing to us.