Which AI Crawlers Visit Your Website, and Should You Block Them?
Published · Last updated · Written and published by Precise Search SEO, Leskovac, Serbia.
AI crawlers fall into three groups by purpose: fetching content for model training, indexing for retrieval inside an AI assistant, and fetching a single page when a user asks for it. Blocking is therefore not one decision but three, because each group carries a different consequence — blocking a retrieval agent removes the possibility of being cited, blocking a training agent does not.
This page separates those three purposes, shows how each is controlled in robots.txt, and states what every decision gains and costs.
Why is blocking not a single decision?
Because the same website is visited by agents with three different purposes, and the consequence of blocking is not the same. Training uses content to train a model and returns no visit. Retrieval indexes content so an assistant can present it as a source. A user-triggered fetch opens exactly the link the user supplied.
- Model training — content enters training and no visit is returned. Blocking costs use in training, not visibility.
- Retrieval in an assistant — content is indexed so it can be presented as a source. Blocking costs the citation and the visit.
- User-triggered fetch — exactly the link the user supplied is opened. Blocking costs a reader who asked for your page by name.
Which agents have public documentation?
The table below lists only agents whose names are published by the company operating them. The exact string and the behaviour rules should always be checked at the source, because the list changes.
| Agent | Operated by | Purpose |
|---|---|---|
GPTBot | OpenAI | Model training |
OAI-SearchBot | OpenAI | Retrieval in assistant |
ChatGPT-User | OpenAI | User-triggered fetch |
ClaudeBot | Anthropic | Model training |
PerplexityBot | Perplexity | Retrieval in assistant |
Google-Extended | Control over use in generative features | |
Applebot-Extended | Apple | Control over use in training |
Meta-ExternalAgent | Meta | Model training |
Amazonbot | Amazon | Training and assistant systems |
Claude-User | Anthropic | User-triggered fetch |
Applebot | Apple | Crawling for search and assistant systems |
BingBot | Microsoft | Crawling for search and assistant systems |
Bytespider | ByteDance | Model training |
CCBot | Common Crawl | Public web archive used by third parties for training |
DuckAssistBot | DuckDuckGo | Retrieval in assistant |
Baiduspider | Baidu | Crawling for search and assistant systems |
The last eight rows were confirmed by logs on our own domains in Cloudflare AI Crawl Control on , rather than copied from somebody else's list. The exact string and behaviour rules should still be checked at the operating company.
Why this table is not exhaustive: the list of AI agents changes faster than any published inventory, and some companies do not publish an agent name at all. Only agents with public documentation at the operating company are listed here. Names circulating on secondary sources are not reproduced without checking at the source — a wrong string in robots.txt blocks nothing while creating the impression that it does.
Google-Extended and Applebot-Extended are not crawlers in the same sense as the others. They are control tokens governing how content may be used, while crawling for conventional search remains with Googlebot and Applebot respectively.
Who actually crawls the site — a snapshot from two domains
Across two of our own projects, within a single day, AI crawlers made 1,164 requests to taxi.co.rs and 618 to leskovac.taxi. Most of that comes from OpenAI agents. The most crawled path on taxi.co.rs is a city fare page; on leskovac.taxi it is the root page.
| Company | taxi.co.rs | leskovac.taxi |
|---|---|---|
| OpenAI | 485 | 155 |
| Microsoft | 220 | 127 |
| 103 | 70 | |
| ByteDance | 61 | 58 |
| Common Crawl | 39 | 9 |
| Meta | not recorded | 33 |
| Amazon | not recorded | 24 |
| Anthropic | 6 | 20 |
| Perplexity | 18 | 19 |
| Apple | 20 | 3 |
| DuckDuckGo | 0 | not recorded |
| Measure | taxi.co.rs | leskovac.taxi |
|---|---|---|
| Total requests | 1,164 | 618 |
| Allowed | 961 | 520 |
| Unsuccessful | 203 | 98 |
| HTTP 200 responses | 786 | 444 |
| Requests from ChatGPT-User | 402 | 98 |
| Most crawled path | /cenovnik/, 83 successful requests | /, 56 successful requests |
What is worth noticing here
On taxi.co.rs the most crawled path is a fare page. That is the same page type holding the highest share of impressions in AI features in Search Console. Two independent sources, measuring two different events, show the same pattern.
A fare page is one whose main content is a table of rates. The same property that makes it extractable in an answer also makes it a more frequent destination for crawling. On leskovac.taxi, where the site is not the primary channel, the most crawled path is the root page rather than a fare page.
What this snapshot is, and what it is not: it covers a single day, not an average and not a trend. Crawling is uneven by nature, so a daily total is not taken as a characteristic volume — which is why the change against the previous day is not quoted here. A crawl and a citation are two separate events and no report links them: logs show that a page was fetched, not that it was used in an answer. Cloudflare data and Search Console data do not share a definition of the event and are not added together.
Source: Cloudflare AI Crawl Control, accounts owned by the author, taxi.co.rs and leskovac.taxi, 24-hour window, read on . Companies with no recorded requests on one domain are marked “not recorded”, meaning they did not appear within that window, not that they are blocked.
How are these agents controlled in robots.txt?
A rule is written per agent, with User-agent and Disallow. An empty Disallow allows everything; Disallow: / denies the whole site to that agent.
An example matching a website whose goal is visibility, but which does not want its content used in model training:
| Decision | Directive | Consequence |
|---|---|---|
| Deny training | User-agent: GPTBot · Disallow: / | Content stays out of training, visibility remains |
| Allow retrieval | User-agent: OAI-SearchBot · Disallow: | The site can be presented as a source |
| Allow user-triggered fetch | User-agent: ChatGPT-User · Disallow: | A user can open the page through the assistant |
The same pattern applies to every agent in the table above, according to which purpose it serves. Rules are written per agent — one block does not cover several names unless they are listed explicitly.
Check before publishing: a misspelled agent name produces no error and no warning. The rule simply never takes effect, while the site owner believes it has. Every string should therefore be copied from the operating company's own page rather than from somebody else's list.
What does blocking cost?
Blocking a retrieval agent costs the possibility of being named as a source in an assistant's answer, and with it the visit that would have followed. Blocking a training agent costs no visibility, but brings no measurable gain either.
| Decision | Gained | Lost |
|---|---|---|
| Block training | Content stays out of model training | Nothing measurable in visibility |
| Block retrieval | Content does not appear in the assistant | Citation, visit, and presence beside competitors who did not block |
| Block user-triggered fetch | Content is not read through the assistant | A user who explicitly asked for your page |
| No blocking | Full eligibility for citation | Control over use in training |
The decision depends on what the product is. If content is a means of reaching work, blocking retrieval works against the goal. If content is itself the product being sold, the reverse holds.
What is not claimed here: we hold no measurement showing the effect of blocking on traffic. The table above describes consequences by agent function, not measured outcomes. Any assessment of whether blocking pays requires the site's own before-and-after data, and even then remains an estimate, because the surrounding environment changes independently.
How do you verify who actually visited?
A user agent string is trivially spoofed, so the log entry alone is not evidence. A visit is confirmed by comparing the IP address against the ranges the company publishes, or through a reverse DNS lookup where one is supported.
- From server or CDN logs, isolate requests by agent name.
- For each name, take the IP addresses and compare them against the operating company's published ranges.
- Keep requests that fail the check separately — that is not the agent it claims to be.
- Track frequency per day rather than per individual visit.
Logs show that a page was fetched. They do not show that it was used in an answer. A crawl and a citation are two separate events and no report links them — what can be measured in AI search at all is set out on the page about preparing a website for AI search, and how the Google report is read on the page about measuring results.
Common questions about AI crawlers
Does blocking AI crawlers harm conventional rankings?
It depends which agent is blocked. Some agents serve model training and others serve retrieval. Blocking a training agent does not touch crawling for conventional search; blocking a retrieval agent removes the possibility of the source being shown and cited.
Do all AI crawlers respect robots.txt?
The published crawlers of large companies state that they respect robots.txt. For agents with no public documentation this cannot be assumed, so control is applied at server or CDN level rather than through robots.txt alone.
How do you verify which crawler actually visited?
A user agent string is trivially spoofed, so a visit is confirmed by checking the IP against the ranges the company publishes, or through a reverse DNS lookup where supported. Without that check, a log entry is not evidence.
Does a crawler visit mean the site will be cited?
No. A crawl and a citation are two separate events and no report links them. Logs show that a page was fetched, not that it was used in an answer.
What is a reasonable default?
For a website whose goal is visibility, a reasonable default is to allow retrieval agents and user-triggered fetchers, and to decide on training agents according to the value of the content. For a website whose content is itself the product, the decision runs the other way.
Primary sources
- OpenAI — crawler documentation
- Google Search Central — overview of Google crawlers
- Google Search Central — AI features and your website
Verify agent names at the operating company before every change to robots.txt. Lists on secondary sources frequently contain strings that were never published.
This guide is a method. How it is carried out on a real website, with a price and a timeline, is shown by the semantic SEO audit.
Not sure who is crawling your website?
Send the URL and access to your logs or CDN report.