Block All AI Bots? You May Be Blocking Your Reach

“We'll just block all AI bots.”

That sounds like a clear privacy or content policy. Technically, it can produce the opposite of clarity: one broad rule can prevent model training, reduce search visibility, interfere with user-initiated agents, and leave nobody able to explain later why the website no longer appears in an AI search product.

This distinction is becoming more visible in 2026 because providers increasingly separate crawlers by purpose. OpenAI, for example, distinguishes OAI-SearchBot for ChatGPT search from GPTBot as a control related to potential model training. Google uses Google-Extended as a separate robots.txt token for certain Gemini uses without changing ordinary Google Search inclusion. Cloudflare now explicitly distinguishes AI Search, Agent, and Training behavior and has announced new defaults for new domains beginning September 15, 2026.

The operational question is no longer:

“Should AI bots access the website?”

It is:

“Which automated systems may retrieve which content for which purpose — and how is that decision actually enforced?”

First misconception: robots.txt is not a firewall

RFC 9309 standardizes the Robots Exclusion Protocol. It defines how website operators can publish rules telling crawlers which URI paths they are requested to access or avoid.

One sentence matters enormously in production: these rules are not access authorization.

If a URL must genuinely remain confidential, internal, or restricted, use authentication, authorization, network controls, WAF rules, or another enforceable security boundary. A line such as Disallow: /internal/ does not stop somebody from requesting the URL with an ordinary HTTP client.

That is true for traditional search engines and AI crawlers alike.

robots.txt is a published preference for compliant crawlers. It is useful — but it is not a lock.

Second misconception: “AI” is one purpose

Automated retrieval now serves several very different functions.

1. Search and citation

A search crawler gathers content so that a page can later be discovered, summarized, cited, or linked in search experiences.

OpenAI currently documents that publishers who want site content included in summaries and snippets in ChatGPT search should not block OAI-SearchBot. OpenAI also notes that the site host or CDN must technically permit traffic from its published crawler infrastructure.

For website owners, that is a reach decision.

If you block OAI-SearchBot, you should not simultaneously assume that content will remain equally discoverable through that crawling path in ChatGPT search.

2. Model training

Training is a different use.

OpenAI identifies GPTBot as the relevant user-agent for publishers that want to exclude sites or pages from potential training.

Google follows a similar separation with a control token of its own: Google-Extended. Google says this token does not affect inclusion in normal Google Search and is not a Google Search ranking signal. Instead, it controls certain uses of content Google already crawls for future Gemini model training and selected grounding functionality.

That distinction creates a useful policy option:

A site can allow search discovery while separately declining model training.

A blanket User-agent: * / Disallow: / destroys that nuance.

3. Agents and user-initiated retrieval

Agents are different again. They may act in real time because a person is actively trying to complete a task: read information, prepare a booking, inspect a page, or understand a form.

Cloudflare now treats Agent activity as a separate behavior category from Search and Training.

That model is useful even if you do not use Cloudflare. A bot acting now on behalf of one user is functionally different from a crawler collecting millions of pages for a future index or training dataset.

When every automated request is labelled “AI bot,” the business decision disappears.

Why this matters right now

Cloudflare has announced new AI-bot defaults for new domains beginning September 15, 2026. Under those defaults, Search bots remain allowed, while bots classified as Training or Agent are blocked on pages displaying ads. Mixed-purpose crawlers that combine search and training receive additional treatment when AI-training blocking is configured.

That is not automatically the right policy for every website.

What matters is the direction of infrastructure design: away from one “AI bot” toggle and toward purpose-specific controls.

For website operators, crawler policy is becoming part of content strategy, SEO, privacy engineering, and platform operations.

Six common red flags

Red flag 1: A blanket block was added “to be safe”

The site has:

User-agent: *

Disallow: /

Or a CDN feature blocks “AI bots” without anybody knowing which behaviors that category includes.

That can be valid when a public website intentionally wants no automated indexing at all. For many business websites, however, it is too coarse.

Ask instead:

• Should the site appear in traditional search?
• Should it be discoverable in AI search and answer systems?
• Should the content be available for model training?
• Should agents acting for a user be able to access it?
• Are there paths that no bot should retrieve?
• Which decisions are content policy and which require technical enforcement?

Red flag 2: robots.txt and the WAF disagree

robots.txt allows a search crawler, but the WAF blocks its traffic with 403.

The published permission is then operationally useless.

The opposite can happen too. robots.txt asks a crawler not to access a path, while the WAF allows the request completely. For compliant crawlers that may be sufficient. For non-compliant scrapers it is not.

An audit must compare both layers:

What are we asking for, and what is the infrastructure actually enforcing?

Red flag 3: Search and training are coupled accidentally

A team wants to decline model training, so it blocks every known AI-related crawler.

That can also block a search crawler that might otherwise generate referral traffic or visibility.

OpenAI and Google now provide separate controls specifically because these are different use cases.

That does not mean every site should allow AI search. It means the decision can be deliberate.

Red flag 4: The policy exists only on the main domain

www.example.com/robots.txt is well maintained. Meanwhile:

• shop.example.com
• docs.example.com
• app.example.com
• help.example.com
• cdn.example.com
• old campaign hosts
• preview and staging systems

have different rules or no policy at all.

RFC 9309 scopes robots.txt to the relevant service or host. A rule on the marketing domain is not a universal company policy.

Red flag 5: Sensitive content is “protected” only by robots.txt

A customer portal, export, private PDF, or staging directory appears under Disallow.

That is not access control.

If content must not be public, protect the response technically with authentication, authorization, private storage URLs, network restrictions, or equivalent controls.

A crawler audit should therefore look for paths where the operator appears to rely on crawl instructions for confidentiality rather than real security.

Red flag 6: Nobody measures the effect

The policy changed six months ago, but nobody reviews:

• crawler logs
• WAF events
• 403 and 429 rates
• referrals from AI search
• search visibility
• new or changed user-agents
• hostnames without robots.txt
• crawlers that continue requesting disallowed paths

At that point the policy is a file, not an operational process.

A practical decision model

Instead of “allow or block AI,” use a small matrix.

Public editorial content

Examples: magazine articles, guides, product pages, service pages.

A possible policy might be:

• traditional search: allow
• AI search: allow
• training: decide separately
• agents: generally allow while controlling abuse and load
• technical rate limits: yes
• sensitive parameters and admin paths: block or protect

Paid or licensed content

Examples: member areas, premium databases, paid reports.

The main boundary here should not be robots.txt. It should be authentication and authorization.

Additional crawler policy may still help, but it does not change the central rule: content behind an entitlement boundary must remain protected even when a bot ignores published preferences.

User-specific or confidential areas

Examples: dashboards, orders, case files, health information, personnel data.

Crawler policy is secondary here. These resources should not be publicly accessible.

If changing the User-Agent string is enough to gain access, the problem is security, not crawler governance.

Public APIs and documentation

For API documentation and developer portals, AI discoverability may be strategically useful. At the same time, aggressive automated access can create API cost or origin load.

That calls for differentiated controls:

• documentation crawlable
• production API authenticated
• demo endpoints rate-limited
• bot traffic measurable
• no secrets in public examples or schemas

How to run an AI crawler audit

1. Inventory every hostname

Do not stop at the marketing domain. Include subdomains, help centers, shops, apps, API documentation, older domains, CDN hosts, and staging.

For each host:

• does /robots.txt exist?
• does it return 200 and text/plain?
• does it contain the intended groups?
• does it conflict with other hosts?

2. Inventory user-agent rules

Document the crawlers that matter to your strategy.

For OpenAI, the policy for OAI-SearchBot may intentionally differ from the policy for GPTBot.

For Google, Google-Extended can be evaluated independently from ordinary search crawling.

Review the list against current provider documentation regularly. User-agents and product boundaries can change.

3. Test real access

A clean robots.txt does not prove that the request succeeds.

Review:

• CDN
• WAF
• bot management
• rate limits
• geo rules
• JavaScript challenges
• CAPTCHAs
• reverse proxy
• origin firewall

A legitimate crawler can be explicitly allowed and still fail at another security layer.

4. Separate preference from enforcement

For every important path, answer:

Is this a request to a compliant crawler, or a real technical block?

That simple distinction prevents some of the most dangerous configurations.

5. Measure outcomes

After changing policy, observe for at least several weeks:

• which crawlers actually arrive
• which paths they request
• which controls block them
• whether referral traffic changes
• whether search or AI visibility changes
• whether origin load rises or falls
• whether legitimate user journeys are harmed by bot protection

What Website-Pflichtencheck would inspect

An AI-crawler and robots.txt review can examine:

• robots.txt across main domains and subdomains
• search, training, and agent policies
• OpenAI- and Google-specific controls
• inconsistencies between robots.txt, WAF, and CDN behavior
• publicly reachable paths incorrectly treated as protected because they are disallowed
• crawler and bot logs
• 403, 429, and challenge behavior
• hostnames without a defined policy
• AI-search referral measurement
• indexing and visibility consequences
• ownership and documentation for crawler-policy changes

The goal is not to be “AI friendly” or “AI hostile.”

The goal is to make a precise technical decision about which uses are wanted and which are not.

A public guide can remain visible in search and AI search while training is declined. A customer portal can block every crawler and still use real authentication. An agent can be allowed to access public information without that becoming a blanket training permission.

In 2026, a single switch labelled “AI bots” compresses several different business decisions into one technical shortcut. That is exactly where misconfiguration begins.