About
Anthropic's web crawler used to train Claude AI models. Fetches publicly available web content to improve Claude's knowledge and capabilities.
Purpose
AI model training data collection
User Agent String
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +https://www.anthropic.com/bot)
How to Control in robots.txt
🚫 Block ClaudeBot
User-agent: ClaudeBot Disallow: /
✅ Allow ClaudeBot
User-agent: ClaudeBot Allow: /
Complete Guide: How to Block ClaudeBot
Server-level blocking, nginx configs, Cloudflare rules, Next.js middleware, and more →
Is ClaudeBot crawling your site?
Run a free scan to check if Anthropic's crawler is accessing your website.
Check if ClaudeBot is crawling YOUR site →