Robots.txt in 2026: What Blocking AI Crawlers Actually Does to Your Visibility

Most business owners touch robots.txt once, for the wrong reason, and then forget it. Someone reads that AI companies are scraping the web, adds four Disallow lines, and assumes the problem is handled.

Two things are usually wrong. The rules often block the wrong bot. And robots.txt is not a lock.

Here is what the file actually does in 2026, what each AI crawler is for, and how to decide what to let in.

Key takeaways

• robots.txt is a request defined by an IETF standard, RFC 9309. Crawlers that honour it treat it as policy, not access control.
• "AI crawler" covers three different jobs: training, search indexing, and fetching a page because a person asked. Blocking one does not block the others.
• Blocking a search bot such as OAI-SearchBot or Claude-SearchBot is the setting most likely to erase you from AI answers, because the vendors say so in their own documentation.
• The ground moved in late 2026: Anthropic began honouring a machine-readable AI-training preference signal on 30 October 2026, and Google says it will support the same signal in early 2027.
• Enforcement stops a bot that ignores you. Declaration keeps the honest ones reading.

What robots.txt is, and what it is not

The Robots Exclusion Protocol was standardised as RFC 9309 in September 2022, with Google engineers among the authors. Section 3 of the standard is blunt: "The Robots Exclusion Protocol is not a substitute for valid content security measures."

That sentence settles most arguments. To protect a members area or a staging server, robots.txt is the wrong tool, and the RFC says so. To state a preference to crawlers that have agreed to read it, it is exactly the right tool.

Two details surprise people. Crawlers are told not to cache the file for more than 24 hours, and both OpenAI and Perplexity document roughly a 24-hour window before their systems reflect a change. Add a rule at 3pm and test at 3.20pm, and you have tested nothing.

Four jobs hiding behind the phrase "AI crawler"

The confusion is not technical. One label covers four unrelated products. Blocking a training crawler is a content-licensing decision. Blocking a search crawler is a visibility decision. The vendors split these on purpose, so you can choose.

For the wider picture of who crawls the web and why, see 210 crawlers: who they are and what they actually do.

| Bot | What it is for | What blocking it actually changes | | --- | --- | --- | | GPTBot | Collects content that may be used to train OpenAI's foundation models | Signals future content should not be used for training. Not the switch that removes you from ChatGPT search | | OAI-SearchBot | Surfaces sites in ChatGPT's search results | OpenAI states that opted-out sites "will not be shown in ChatGPT search answers", though they may still appear as navigational links | | ChatGPT-User | Fetches a page because a person asked about it | User-initiated, so OpenAI notes robots.txt rules "may not apply" | | Googlebot | Crawls and indexes for Google Search | Removes you from Google Search entirely. Never touch this to "opt out of AI" | | Google-Extended | A control token, not a separate crawler. Governs Gemini training and Gemini grounding | Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" | | ClaudeBot | Collects content that may contribute to training Anthropic's models | Signals that future material should be excluded from training datasets | | Claude-SearchBot | Navigates the web to improve Claude's search quality | Anthropic warns this "may reduce your site's visibility and accuracy in user search results" | | Claude-User | Fetches pages during user-initiated requests | Controls which sites can be reached through user questions | | PerplexityBot | Surfaces and links sites in Perplexity results | Removes you from Perplexity's search results | | Perplexity-User | Fetches a page in response to a user question | Perplexity states this fetcher "generally ignores robots.txt rules" |

All four vendors publish their own definitions, and they are worth reading directly rather than through a summary: OpenAI crawlers, Google's common crawlers, Anthropic's crawler policy and Perplexity's crawler documentation.

Two subtleties are easy to miss. Google-Extended does not exist as a separate HTTP request: Google crawls with its existing user agents and reads that token as a control. And Perplexity's user-triggered fetcher openly says it ignores robots.txt, which means the only real control there is server-side.

The most expensive mistake

The costly error is not blocking AI. It is blocking the wrong half and calling it a strategy.

A Lagos interior studio decides to protect its portfolio images and pastes a blanket block for every AI user agent it can find. Nothing appears to break. Over the following months its pages stop being the source answer engines pull from when someone asks for a furniture maker in Lekki. The site still ranks for the old queries, and is quietly absent wherever a model composes an answer.

The same failure shows up in a Nairobi clinic that blocked "all AI bots" to stop patient questions being scraped, and in a Manchester accountant who copied a blocklist from a media company's blog. Publishers block aggressively because they sell content licences. A five-page service business has nothing to license and everything to lose.

The reverse error is just as common: a twelve-year-old file, still disallowing /wp-admin/ on a site that no longer runs WordPress, or disallowing a path that now holds the product pages.

What changed in October 2026

For years the only signal available was block the bot or don't. That changed at the end of October.

Anthropic states that from 30 October 2026 it began recognising a machine-readable signal letting site owners express a preference about AI training, following a draft from the IETF's aipref working group. Their documentation also explains precedence: rules naming ClaudeBot directly are treated as more specific than the general signal, so an explicit ClaudeBot rule wins.

Google says it intends to support the same signal, with the token user-agent: __usage_ai-training, in early 2027, as noted in its common crawlers documentation.

The practical consequence is that the file is becoming a two-dial setup: a general training preference, plus named per-bot overrides. It also means a general token is not a reliable way to exclude one specific crawler. If precision matters, name the bot.

A 20-minute audit you can run this week

1. Open yourdomain.com/robots.txt and read the whole file. Write down every AI user agent it names.
2. Classify each rule as training, search, or user-triggered. If you cannot tell, you are not ready to keep the rule.
3. Keep the search bots allowed on every page that earns money. OAI-SearchBot and PerplexityBot are what put you in front of a person asking a question.
4. To restrict training, name only the training bots, GPTBot and ClaudeBot, instead of using a wildcard.
5. Decide per section, not per site. A pricing page and a media library do not deserve the same rule.
6. Enforce where it matters. A WAF or bot-management rule is what actually blocks a crawler; robots.txt only asks.
7. Re-check in 48 hours, not two minutes, then leave it alone for a quarter.

Then do the step almost nobody does: check whether the pages you care about can be read at all. If they only exist after JavaScript runs, the robots.txt debate is academic, as covered in why daily submission beats one-off submission.

What this will not do

Robots.txt is worth understanding mainly for its limits.

It does not guarantee a citation, a ranking, or an appearance in any AI answer. Being crawlable and being quotable are different problems, and only the first lives in this file.

It does not remove content already collected. An opt-out today governs future crawling.

It does not stop a user-triggered fetch, because several honest vendors will tell you they ignore the file when a person asked for the page.

And it is not a security boundary. The RFC says that plainly, and so does every incident where a "disallowed" path turned out to be publicly readable.

FAQ

Does blocking GPTBot remove my site from ChatGPT? No. GPTBot is the training crawler. OAI-SearchBot is the one tied to appearing in ChatGPT's search answers, and OpenAI treats the two settings as independent.

Is robots.txt legally binding? It is a standard, not a contract. Compliance is voluntary, which is why vendors publish statements saying they honour it, and why enforcement still lives at the server.

Should a small business block AI crawlers at all? Block training if you have content worth licensing. Keep search and answer crawlers allowed if you want to be findable. Those are two decisions, and most businesses only ever make one.

The part that belongs on a schedule

A robots.txt that is correct in November tends to be wrong by spring, because you add a section, launch a page, or a vendor renames a bot. The useful discipline is not a perfect initial file. It is knowing what yours says, and re-reading it when the site changes.

Knowing which bots exist is the first half of visibility. The second half is making sure the ones that matter can reach fresh, readable pages. If you want the crawl and submission side handled daily while you get on with the business, tsboi.ai is built for exactly that. It does not promise rankings or citations. It removes the technical reasons a crawler would skip you.