How to Block xAI-Bot

What Does xAI-Bot Collect?

xAI-Bot crawls publicly available web pages to build training data for Grok, xAI's AI assistant embedded across X (formerly Twitter) and available via grok.com. It launched publicly in early 2024 and quickly became one of the faster-growing AI training crawlers.

Unlike other AI companies whose assistants are general-purpose, Grok is positioned as a real-time knowledge engine — its core differentiator is knowing about recent events. This makes xAI-Bot unusually aggressive on news sites, commentary platforms, industry blogs, and any content that covers current topics. If your site publishes timely content, xAI-Bot has almost certainly visited it.

xAI draws from two data sources: direct X/Twitter post data (which it has privileged access to as the parent company) and external web crawling via xAI-Bot. Blocking xAI-Bot addresses the web crawl channel. Content that users post about your site on X is a separate pipeline that robots.txt cannot control.

Option 1: Block via robots.txt (Recommended)

User-agent: xAI-Bot
Disallow: /

# Block xAI-Bot from articles and premium content
User-agent: xAI-Bot
Disallow: /articles/
Disallow: /news/
Disallow: /premium/
Disallow: /members/

# Block all major AI training crawlers
User-agent: xAI-Bot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Normal search indexing — leave these alone
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Option 2: Next.js App Router

Generate your robots.txt programmatically from app/robots.ts:

import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: 'xAI-Bot', disallow: ['/'] },
      { userAgent: 'GPTBot', disallow: ['/'] },
      { userAgent: 'ClaudeBot', disallow: ['/'] },
      { userAgent: 'anthropic-ai', disallow: ['/'] },
      { userAgent: 'Google-Extended', disallow: ['/'] },
      { userAgent: 'PerplexityBot', disallow: ['/'] },
      { userAgent: 'CCBot', disallow: ['/'] },
      { userAgent: 'Bytespider', disallow: ['/'] },
      { userAgent: 'Googlebot', allow: ['/'] },
      { userAgent: '*', allow: ['/'] },
    ],
    sitemap: 'https://yoursite.com/sitemap.xml',
  };
}

Option 3: nginx — Hard 403 Block

Since xAI-Bot reliably respects robots.txt, a server-level block is optional. Use it if you want guaranteed enforcement regardless of robots.txt, or to reduce server load from crawler requests entirely.

# In your server {} block
if ($http_user_agent ~* "xAI-Bot") {
    return 403;
}

Returns HTTP 403 before the request reaches your application. Combine with robots.txt for belt-and-suspenders protection.

Option 4: Cloudflare WAF Rule

If your site is behind Cloudflare, create a custom WAF rule to block xAI-Bot at the network edge:

(http.user_agent contains "xAI-Bot")

Set the action to Block. This drops requests before they hit your server — zero load, zero logging overhead.

Navigate to: Cloudflare Dashboard → Security → WAF → Custom Rules → Create rule

Verify Your Block

To confirm xAI-Bot is seeing your robots.txt correctly, check your access logs for its user agent string:

# Check nginx access logs for xAI-Bot
grep "xAI-Bot" /var/log/nginx/access.log | tail -20

# Check if it's fetching robots.txt (good sign)
grep "xAI-Bot" /var/log/nginx/access.log | grep "robots.txt"

# Confirm blocked requests (nginx hard block)
grep "xAI-Bot" /var/log/nginx/access.log | grep " 403 "

Seeing xAI-Bot fetch /robots.txt followed by no requests to your content means the block is working. If you see xAI-Bot on content pages after adding the robots.txt rule, add a server-level block as a backup.

Should You Block xAI-Bot?

The right answer depends on your goals. Here's the honest tradeoff:

Frequently Asked Questions

Related Guides