How is this different from Google Analytics?

Google Analytics shows you traffic. Shadow shows you traffic, AI bot activity, what AI platforms say about your brand, AND tells you what to do about all of it. It's analytics + AI intelligence + action steps in one tool.

Do I need to install anything?

For basic monitoring (bot detection, AI perception, readiness score) — nope, just enter your URL. For full visitor analytics (clicks, behavior, sessions), add one script tag. One-click integrations for Vercel, Shopify, WordPress, and more.

Will it slow down my site?

No. The script is under 5KB and loads async. Zero impact on page speed or Core Web Vitals. External monitoring has literally no impact — it watches from the outside.

What AI bots does Shadow detect?

All of them. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, Amazonbot, and dozens more. The Shadow Network means new bots get identified across all users instantly.

What do you mean by "actionable steps"?

Shadow doesn't just show you graphs. It says things like: "ChatGPT has your pricing wrong — add structured data to /pricing to fix it" or "Your bounce rate on /features is 68% — here's why and what to change." Specific, do-it-today recommendations.

Can Shadow block bots?

Shadow is a telescope, not a shield. It shows you who's visiting and what AI says about you. It generates block rules and robots.txt configs you can apply — but it doesn't intercept traffic.

Yes. Shadow never collects PII. IP addresses are hashed after classification. No cookies on your visitors. All Shadow Network data is anonymized. GDPR compliant by design.

What is OAI-SearchBot?

OAI-SearchBot is OpenAI's web crawler for ChatGPT Search (formerly SearchGPT). It crawls and indexes web pages so that ChatGPT can surface them as search results when users ask questions. It is completely separate from GPTBot, which collects training data, and ChatGPT-User, which fetches specific URLs in real-time at a user's request.

Does blocking OAI-SearchBot stop ChatGPT from training on my content?

No. OAI-SearchBot and GPTBot serve entirely different purposes. GPTBot collects training data for OpenAI's language models. OAI-SearchBot indexes content for ChatGPT's search feature. Blocking OAI-SearchBot removes your site from ChatGPT Search results — it has no effect on training data collection. To stop training, you need to block GPTBot separately.

If I block OAI-SearchBot, will my site disappear from ChatGPT Search entirely?

Largely yes, for proactive indexing. ChatGPT Search relies on OAI-SearchBot's index as its primary source. Blocking it prevents your content from being added to or refreshed in that index. However, ChatGPT may still surface your site via third-party search data it licenses (e.g., Bing). To comprehensively opt out of ChatGPT Search, block OAI-SearchBot and optionally opt out of Bing's index.

What is the difference between OAI-SearchBot, GPTBot, and ChatGPT-User?

OpenAI operates three distinct bots: (1) GPTBot — crawls the web to collect training data for OpenAI's AI models. Blocking it prevents your content from being used in future model training. (2) OAI-SearchBot — crawls the web to build and maintain the search index for ChatGPT Search. Blocking it removes your site from ChatGPT Search results. (3) ChatGPT-User — not a crawler; it's the user agent used when a ChatGPT user asks the chatbot to browse a specific URL in real-time. You can control each independently in robots.txt.

Should I block OAI-SearchBot?

It depends on your goals. Block OAI-SearchBot if: you have a paywalled or subscription site and don't want free summaries in ChatGPT, you've seen your traffic cannibalized by AI Overviews/ChatGPT answers, or your content is a competitive moat you don't want indexed. Allow OAI-SearchBot if: you want discovery from the growing base of ChatGPT Search users, your content benefits from broader distribution, or you're a publisher / news site seeking AI search traffic.

Does OpenAI respect robots.txt for OAI-SearchBot?

Yes. OpenAI has publicly documented that OAI-SearchBot respects robots.txt Disallow directives. Independent testing has confirmed compliance. A robots.txt block is sufficient for most publishers to opt out of ChatGPT Search indexing.

Can I block GPTBot but allow OAI-SearchBot (or vice versa)?

Yes, and this is actually a common configuration. Many publishers want to prevent training data collection (block GPTBot) while still benefiting from ChatGPT Search traffic (allow OAI-SearchBot). Since they are separate user agents, you can configure each independently in robots.txt with individual User-agent blocks.

OpenAIRespects robots.txtSearch Index

How to Block OAI-SearchBot: Control Your Presence in ChatGPT Search

OAI-SearchBot is OpenAI's crawler for ChatGPT Search. It's completely separate from GPTBot. Blocking it removes your site from ChatGPT's search results — not training data. Here's what that means and when it makes sense.

Updated March 2026

OAI-SearchBot ≠ GPTBot ≠ ChatGPT-User

OpenAI operates three distinct bots. Blocking one does not affect the others. Most publishers conflate them — here's the real breakdown:

GPTBotAutonomous training crawler. Collects web data to train future GPT models.

OAI-SearchBotSearch index crawler. Builds and maintains the ChatGPT Search index (formerly SearchGPT).

ChatGPT-UserOn-demand browsing agent. Fetches a specific URL only when a ChatGPT user asks it to read a page.

What Does OAI-SearchBot Actually Do?

OAI-SearchBot is OpenAI's web crawler dedicated to building and maintaining the search index that powers ChatGPT Search (previously marketed as SearchGPT). When a ChatGPT user asks a question that triggers a web search, the results come from this index.

Unlike GPTBot, which crawls to collect training data for future GPT models, OAI-SearchBot crawls to keep the ChatGPT Search index fresh and comprehensive. It behaves like a traditional search engine crawler — following links, indexing page content, and refreshing previously crawled pages on a regular cadence.

The user agent string is: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); OAI-SearchBot/1.0; +https://openai.com/searchbot

How to Block OAI-SearchBot in robots.txt

Add this to your robots.txt to prevent OAI-SearchBot from indexing your site for ChatGPT Search:

robots.txtBlock OAI-SearchBot only

User-agent: OAI-SearchBot
Disallow: /

Common configuration: block training but allow ChatGPT Search visibility:

robots.txtBlock training, allow search

# Block training data collection
User-agent: GPTBot
Disallow: /

# Allow ChatGPT Search indexing
User-agent: OAI-SearchBot
Allow: /

# Block real-time browsing
User-agent: ChatGPT-User
Disallow: /

Important: Each bot needs its own directive

You cannot combine multiple user agents under one block in robots.txt. Each OpenAI crawler must have its own User-agent: section.

Who Should Block OAI-SearchBot?

Blocking OAI-SearchBot is a trade-off: you lose ChatGPT Search visibility but prevent your content from being summarized in AI search results. Consider your situation:

🔒

Paywalled publishers

If ChatGPT Search summarizes your premium content for free, subscribers have less reason to pay. Blocking prevents this.

📉

Traffic cannibalization concerns

If AI-generated answers are providing your content without a click-through, you lose the visit, the ad revenue, and the conversion.

🛡️

Competitive moat protection

Proprietary research, original analysis, or unique data you don't want surfaced in AI search for competitors to access.

What Blocking OAI-SearchBot Does (and Doesn't) Do

What it stops

• Your content appearing in ChatGPT Search results
• New pages being added to OpenAI's search index
• Existing indexed pages being refreshed

What it doesn't stop

• GPTBot training crawls (block GPTBot separately)
• ChatGPT-User on-demand browsing (block separately)
• Previously indexed pages (they age out over time)
• ChatGPT Search surfacing your content via licensed Bing data

The Bing data feed caveat

ChatGPT Search supplements its own index with licensed data from Bing. Even if you block OAI-SearchBot, your content may still appear in ChatGPT Search results via Bing's index. To comprehensively opt out, you would also need to block Bingbot — which removes you from Bing Search as well.

Verify Your Block Is Working

After deploying your robots.txt changes, verify them at Open Shadow's robots.txt checker. Enter your domain and confirm OAI-SearchBot shows as blocked.

You can also check your server logs for the OAI-SearchBot user agent:

nginx log grep

grep "OAI-SearchBot" /var/log/nginx/access.log

Frequently Asked Questions

Can I block GPTBot but allow OAI-SearchBot?

Yes — this is the most popular configuration. Many publishers want to prevent their content from being used to train GPT models (block GPTBot) while still appearing in ChatGPT Search results where they might receive traffic (allow OAI-SearchBot). Each bot has its own User-agent block in robots.txt.

How long until my site disappears from ChatGPT Search after blocking?

Existing indexed pages don't disappear immediately. They age out of the index over time — typically weeks to months. There is no equivalent of Google Search Console's URL removal tool for ChatGPT Search. Blocking only prevents new or refreshed indexing.

Does blocking OAI-SearchBot affect Google or Bing rankings?

No. OAI-SearchBot is completely separate from Googlebot and Bingbot. Blocking it has zero effect on your traditional search engine rankings or visibility.

Will ChatGPT Search traffic become significant enough to care about?

It depends on your audience. As of early 2026, ChatGPT has 400M+ weekly active users and ChatGPT Search usage is growing rapidly. For some publishers, it's already a meaningful referral source. The decision to block is a bet on whether AI search traffic will replace or supplement traditional search traffic for your site.

Related Guides

How to Block GPTBot

OpenAI's training crawler — different purpose

How to Block ChatGPT-User

OpenAI's browsing crawler

How to Block PerplexityBot

Another AI search engine

robots.txt for AI Bots (Complete Guide)

51+ crawlers, full reference table

Is your site protected from AI bots?

Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.