How to Block AI Training Without Blocking Google Search: New robots.txt AI Opt-Out Explained

Website owners may soon have a much simpler way to say:

“You can index my website—but don’t use my content to train generative AI models.”

On October 30, 2026, a new proposal introduced a shared robots.txt policy signal called:

__usage_ai-training

The idea is simple.

Instead of website owners maintaining separate AI-training rules for OpenAI, Google, Microsoft and other AI companies, participating providers could recognize one common signal.

A whole-site opt-out would look like this:

User-Agent: __usage_ai-training
Disallow: /

That tells participating AI providers that the website owner expresses a preference against using content from the site for generative-AI training.

But there is one extremely important detail:

this does not block Google Search crawling.

The new signal is designed to express a content-usage preference, not a normal crawler-access rule.

That means a website can potentially continue appearing in:

• Google Search;
• Bing Search;
• traditional search results;

while separately saying:

do not use this content to train generative AI models.

For publishers, bloggers, WordPress owners and businesses concerned about AI scraping, this could become one of the most important robots.txt changes in years.

However, the new signal is still being rolled out.

As of October 31, 2026, it is an:

Internet-Draft

rather than a finalized IETF standard.

Here is exactly how it works, what you can do today, and how to block AI training without accidentally damaging your SEO.

What Is the New AI Training robots.txt Signal?

The proposed signal is:

User-Agent: __usage_ai-training

It appears inside a website’s standard:

robots.txt file.

A website owner can then use familiar Allow and Disallow rules.

For example:

User-Agent: __usage_ai-training
Disallow: /

This expresses a preference against AI training across the entire website.

The draft also allows path-specific rules.

Example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That means:

• don’t use most of the site for AI training;
• but AI training is permitted for content under /press-kit/.

The proposal intentionally uses existing robots.txt syntax instead of inventing an entirely new file format.

What Does __usage_ai-training Actually Mean?

Under the draft, the token represents a preference about:

generative AI model training.

The draft defines AI training broadly as using an asset to:

modify the learned parameters of a generative AI model.

That distinction is important.

It does not automatically mean:

• block crawling;
• block indexing;
• block search engines;
• block users;
• block all AI-generated answers.

The signal is about:

how retrieved content may be used for model training.

That is different from simply controlling whether a bot may visit the page.

Does Blocking AI Training Block Google Search?

No—not when implemented correctly.

Google treats its normal search crawler:

Googlebot

separately from its AI-training control:

Google-Extended.

Google states that using Google-Extended does not affect:

• inclusion in Google Search;
• normal Google Search rankings.

So a website owner can already block Google from certain generative-AI uses while continuing to allow Googlebot to crawl and index the website.

The new shared AI-training signal follows the same basic principle:

search crawling and AI-training preference are separate controls.

Why This Matters for Publishers

Until now, AI companies have used different crawler names and control mechanisms.

Website owners may encounter names such as:

• GPTBot;
• OAI-SearchBot;
• Google-Extended;
• Claude-related crawlers;
• Bingbot;
• other AI-provider agents.

That creates a maintenance problem.

A publisher who wants to allow:

search indexing

but block:

AI model training

may have to understand the purpose of each crawler individually.

The shared __usage_ai-training signal is intended to reduce that complexity.

Eventually, one rule could communicate the same training preference to multiple participating AI providers.

Is the New AI Opt-Out Already Working?

Not everywhere.

This is the most important warning in this article.

The proposal was published on:

October 30, 2026.

It is still an:

IETF Internet-Draft.

Internet-Drafts are works in progress.

They can be:

• updated;
• replaced;
• modified;
• abandoned;
• eventually standardized.

The current draft is not itself a finalized internet standard.

So website owners should not assume adding the shared token today instantly changes how every AI company treats their website.

When Will OpenAI Support It?

OpenAI announced that it plans to honor the shared AI-training preference signal.

The company says implementation is expected:

in the coming months.

OpenAI also says its existing crawler controls will continue to work.

That means website owners who already block:

GPTBot

should not remove that rule simply because the new shared token has been proposed.

At least during the transition, existing provider-specific rules remain useful.

When Will Google Support It?

Google says it plans to support the shared signal in:

early 2027.

Google currently provides its own control called:

Google-Extended.

Google says that when it begins honoring the shared signal, a training opt-out using the common token will generally be interpreted similarly to a Google-Extended disallow unless the site explicitly provides a separate Google-Extended instruction.

That means website owners will still be able to override the general signal for Google specifically.

When Will Microsoft Support It?

Microsoft has announced a specific implementation date:

February 1, 2027.

Microsoft says it will begin honoring the shared AI-training preference as it crawls websites.

Microsoft is also introducing additional page-level controls for:

• AI training;
• AI grounding.

Those concepts are separate.

Training affects how content may help build future AI models.

Grounding affects how content may be used to help produce AI-generated answers at request time.

What About Anthropic?

The Internet-Draft was authored by:

Kevin Kelley of Anthropic.

That shows Anthropic is directly involved in the standards effort.

However, website owners should still check Anthropic’s current crawler documentation before assuming a particular implementation date or behavior.

The safest approach remains:

use provider-specific controls until official shared-signal support is confirmed.

What Website Owners Can Do Today

If you want to reduce AI-training use today, the best approach is:

keep existing provider-specific controls and prepare for the shared signal.

You can add the new shared signal if you want to express your preference early, but do not rely on it as your only protection yet.

How to Block OpenAI Training but Allow ChatGPT Search

OpenAI currently separates its search crawler from its training crawler.

The two important user agents are:

GPTBot

and:

OAI-SearchBot.

GPTBot is associated with content that may be used to help train OpenAI’s generative AI foundation models.

OAI-SearchBot is used to help surface websites in ChatGPT search results.

So a website owner could use:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

This tells OpenAI:

do not use GPTBot for training-related crawling

while still allowing:

ChatGPT Search crawling.

That can help preserve potential visibility in ChatGPT search while expressing an opt-out from training use.

Does Blocking GPTBot Remove You From ChatGPT Search?

No.

OpenAI documents GPTBot and OAI-SearchBot as separate controls.

If you block GPTBot but allow OAI-SearchBot, OpenAI says the site can still remain eligible to appear in ChatGPT search results.

This distinction is especially important for publishers.

You may want:

AI referral traffic

without:

AI training use.

OpenAI’s separate crawler architecture allows that choice.

How to Block Google AI Training but Keep Google Search

Google currently provides:

Google-Extended.

A website could use:

User-agent: Google-Extended
Disallow: /

while leaving Googlebot unblocked.

Google states that Google-Extended does not affect:

• inclusion in Google Search;
• Google Search ranking.

That means the site can continue to be indexed by normal Google Search.

Important Google Gemini Trade-Off

There is one important trade-off.

Google-Extended is not only related to future Gemini model training.

Google also uses the control for certain:

Gemini grounding uses.

Grounding means using current web content to help an AI system answer a user’s question.

Google says websites that specifically allow Google-Extended can be eligible to appear in grounded Gemini Apps responses and receive clicks from those experiences.

So a website owner should understand the difference:

Blocking Google-Extended does not block Google Search.

But it may reduce eligibility for some Gemini-grounded experiences.

Google Search vs. Gemini Is Not the Same Thing

This distinction is increasingly important.

A website can potentially participate in:

Google Search

without participating in:

certain Gemini AI uses.

Google Search is driven by Googlebot and Google’s search indexing systems.

Google-Extended is a separate control.

That means SEO and AI-training preferences do not have to be an all-or-nothing decision.

What Will the Shared Rule Look Like?

For a whole-site training opt-out:

User-Agent: __usage_ai-training
Disallow: /

For only one section of a website:

User-Agent: __usage_ai-training
Disallow: /members/

For blocking everything except a public media section:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The draft uses the same path-matching structure already familiar from robots.txt.

Does User-agent: * Block AI Training Under the New Proposal?

No—not automatically.

This is a technical detail website owners should understand.

The draft specifically says wildcard groups such as:

User-agent: *

should not be interpreted as matching:

__usage_ai-training

So if you want to express the new training preference, you need the specific policy token.

For example:

User-Agent: __usage_ai-training
Disallow: /

A generic wildcard rule is not the same thing.

Is __usage_ai-training Really a Crawler Name?

Not exactly.

It appears in the:

User-Agent

field, which normally contains crawler names.

But the draft calls it a:

policy token.

That means it represents a usage purpose rather than an actual crawler identity.

This is a subtle but important distinction.

The token does not necessarily tell a crawler:

“You may not visit this URL.”

It says:

“The website owner expresses a preference against this content being used for AI training.”

Does This Physically Stop an AI Bot From Crawling?

Not necessarily.

The proposal explicitly separates:

crawling behavior

from:

content-usage preference.

A provider may still crawl content for another permitted purpose.

For example:

• search indexing;
• security;
• live grounding;
• other functions.

The important question becomes:

what is the content used for after it is retrieved?

robots.txt Is Not a Security System

Website owners should not treat robots.txt as a technical security barrier.

robots.txt is a publicly readable instruction file.

Responsible crawlers generally respect it.

A malicious crawler can ignore it.

Therefore, robots.txt should never be used to protect:

• passwords;
• private customer information;
• confidential documents;
• internal files;
• paid content that must remain inaccessible.

If content truly must be private, use actual access controls such as:

• authentication;
• authorization;
• server-side restrictions.

Does the AI Opt-Out Stop Web Scraping?

Not universally.

The new signal expresses a preference.

It does not technically prevent a scraper from downloading publicly accessible pages.

A company that chooses to honor the signal may change how it uses that content.

A malicious or non-compliant scraper could ignore it.

That is one reason website owners should distinguish between:

policy preference

and:

technical enforcement.

Can I Block AI Training on Only Some Articles?

Yes.

One advantage of the proposal is:

path-level control.

Suppose your website has:

• normal blog content;
• a press kit;
• paid research.

You could write:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That expresses:

• no training on the general site;
• training permitted under /press-kit/.

You could also block only a specific directory.

Example:

User-Agent: __usage_ai-training
Disallow: /premium-research/

Can Individual Pages Override the robots.txt Rule?

The shared draft itself focuses on robots.txt path rules.

Microsoft has separately announced page-level AI-training controls.

For Bing/Microsoft systems, that creates more granular options.

A website could potentially set a broad robots.txt preference and then use page-level settings for exceptions.

Because implementations vary by provider, website owners should check each provider’s documentation before mixing multiple rules.

What Is AI Grounding?

AI grounding is different from training.

AI training

Content helps modify or create the model itself.

AI grounding

Content is retrieved at query time to help answer a user’s question.

For example, an AI assistant might search the web for:

“What is the 2027 Social Security COLA?”

and use a recent webpage to support its answer.

That is grounding.

The website did not necessarily train the underlying model.

The page was used as a current source.

Why Publishers May Want Grounding but Not Training

This is likely to become a major publisher strategy.

A website owner may say:

I don’t want my articles permanently incorporated into future model training.

But at the same time:

I do want AI assistants to cite my articles and send me traffic.

Those goals are not contradictory.

That is why separate controls for:

• training;
• search;
• grounding;

are becoming important.

Should Bloggers Block AI Training?

There is no universal answer.

Blocking training can make sense if your priority is:

• protecting proprietary content;
• limiting AI reuse;
• maintaining control over premium research;
• enforcing editorial policy.

Allowing training may make sense if your priority is:

• broad distribution;
• participation in AI ecosystems;
• maximizing exposure;
• supporting model development.

The decision depends on your business model.

Should SEO Websites Block AI Training?

For SEO-focused publishers, the more important question is:

Will blocking training reduce traffic?

For Google Search specifically, blocking Google-Extended does not affect normal Google Search indexing or rankings.

For OpenAI, blocking GPTBot while allowing OAI-SearchBot preserves eligibility for ChatGPT Search.

So technically, publishers can separate:

search discoverability

from:

training permission.

However, blocking certain AI-use controls may reduce visibility in specific AI-grounded products.

That trade-off deserves consideration.

Example robots.txt Setup for a Publisher

A publisher who wants to:

• stay in Google Search;
• stay eligible for ChatGPT Search;
• block OpenAI training;
• block Google’s current AI-training control;
• express the future shared AI-training opt-out;

could eventually use a structure similar to:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Google-Extended
Disallow: /

User-Agent: __usage_ai-training
Disallow: /

Important:

do not copy this blindly.

Your robots.txt file may already contain important WordPress, plugin, CDN or crawler rules.

Always review the full file before editing it.

How to Find Your robots.txt File

For most websites, the file is available at:

yourdomain.com/robots.txt

WordPress sites may generate the file automatically.

Depending on your setup, it may be controlled by:

• WordPress;
• your SEO plugin;
• your hosting provider;
• Cloudflare;
• another security or optimization service.

Before editing, determine which system actually manages your robots.txt output.

How to Edit robots.txt in WordPress

The method depends on your website.

Common options include:

• SEO plugin settings;
• hosting control panel;
• server file manager;
• theme or custom plugin code;
• virtual robots.txt filters.

After editing, open your robots.txt URL directly and verify that the new rules actually appear.

Do Not Accidentally Block Googlebot

One of the biggest SEO mistakes would be adding:

User-agent: *
Disallow: /

That can block normal search crawling.

That is very different from:

User-Agent: __usage_ai-training
Disallow: /

or:

User-agent: Google-Extended
Disallow: /

The first generic wildcard rule controls crawling broadly.

The other rules target specific AI-use preferences.

Does noindex Block AI Training?

noindex serves a different purpose.

A noindex directive tells supported search engines not to show a page in search results.

That is not the same thing as an AI-training preference.

If you want:

the page in Google Search

then adding noindex would be the wrong solution.

This is why the new AI-training control matters.

It aims to let websites say:

keep indexing me, but don’t train on me.

Will Blocking AI Training Hurt Google Rankings?

Google says its Google-Extended control does not affect:

Google Search inclusion

or:

Google Search ranking signals.

The proposed shared training token is also designed to remain separate from traditional search crawling.

So there is currently no official basis to say a training opt-out should cause a normal Google ranking penalty.

Could Blocking AI Training Hurt AI Referral Traffic?

Potentially, depending on the provider and product.

For example, Google says explicitly allowing Google-Extended can make a website eligible for grounding in Gemini Apps.

A website that opts out may therefore reduce its visibility in certain Gemini experiences.

Similarly, other providers may separate:

• training;
• search;
• grounding;

in different ways.

Website owners should evaluate each surface rather than assuming all AI traffic works the same way.

What Should Website Owners Do Right Now?

As of October 31, 2026:

Keep your existing crawler rules

Do not remove working GPTBot, Google-Extended or other provider-specific rules yet.

Consider adding the shared preference

You can add the new policy token to express your future preference:

User-Agent: __usage_ai-training
Disallow: /

But remember that support is still rolling out.

Keep Googlebot allowed

If you want Google Search traffic, do not accidentally block Googlebot.

Keep OAI-SearchBot allowed if you want ChatGPT Search visibility

OpenAI separates search from training.

Review AI-grounding consequences

Blocking certain provider controls may also affect visibility in AI-generated answers.

Recheck documentation in 2027

Implementation details can change as the IETF draft evolves and providers complete their rollouts.

Shared AI Training Signal: Current Rollout

Provider	Shared signal status
OpenAI	Plans support in coming months
Google	Plans support in early 2027
Microsoft	Starts honoring Feb. 1, 2027
Anthropic	Participating in standards work; verify final implementation details
IETF status	Internet-Draft, not final standard

These details are current as of:

October 31, 2026.

Frequently Asked Questions

What is __usage_ai-training?

It is a proposed shared robots.txt policy token that lets website owners express whether content should be used to train generative AI models.

How do I block AI training on my entire site?

The proposed syntax is:

User-Agent: __usage_ai-training
Disallow: /

However, provider support is still rolling out.

Will this block Google Search?

No.

The shared training preference is designed separately from normal search crawling.

Can I block Google AI training but stay in Google Search?

Yes.

Google currently supports Google-Extended as a separate AI-use control, and Google says it does not affect normal Search inclusion or rankings.

Can I block OpenAI training but still appear in ChatGPT Search?

Yes.

OpenAI says GPTBot and OAI-SearchBot are separate controls.

You can disallow GPTBot while allowing OAI-SearchBot.

Is the new rule already active?

Not universally.

The shared signal was proposed on October 30, 2026 and is still being implemented.

Is this an official IETF standard?

No.

It is currently an Internet-Draft.

Can a crawler ignore robots.txt?

Yes.

robots.txt is not a security barrier.

Responsible crawlers may honor it, but malicious scrapers can ignore it.

Does this stop all AI scraping?

No.

The signal expresses a training preference.

It does not physically prevent every crawler from accessing public content.

Does it block AI answers?

Not necessarily.

AI grounding and AI training are separate concepts.

A provider may use content in a live answer without using it to train the model.

Can I block only part of my site?

Yes.

The draft supports path-level rules.

Should WordPress bloggers add the rule now?

You can add it to express your preference early, but existing provider-specific controls should generally remain until broad implementation is confirmed.

Does blocking Google-Extended hurt rankings?

Google says Google-Extended does not affect Google Search rankings.

Does blocking GPTBot hurt ChatGPT Search?

OpenAI says search is handled separately through OAI-SearchBot.

Can I allow AI training for one folder?

Yes.

For example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

What is AI grounding?

Grounding means using current web content at answer time to improve an AI response.

It is different from using content to train the model.

Bottom Line

Website owners are finally getting closer to a simple answer to one of the biggest questions created by generative AI:

Can I stay visible in search without allowing my content to train AI models?

The answer is increasingly:

yes.

The new proposed robots.txt signal:

User-Agent: __usage_ai-training
Disallow: /

is designed to let publishers express a broad preference against generative-AI training without changing ordinary search-crawler rules.

But the rollout is not complete.

As of October 31, 2026:

• the signal is still an IETF Internet-Draft;
• OpenAI plans support in the coming months;
• Google plans support in early 2027;
• Microsoft begins honoring it February 1, 2027.

In the meantime, website owners should continue using provider-specific controls.

OpenAI already allows publishers to separate:

GPTBot for training

from:

OAI-SearchBot for ChatGPT Search.

Google already allows publishers to use:

Google-Extended

without affecting normal Google Search indexing or rankings.

For publishers, the larger shift is clear.

The web is moving toward separate choices for:

search

AI grounding

and:

AI model training.

That means the future may no longer require a website owner to choose between:

being discoverable

and:

controlling how AI companies use their content.