About
Bright Data's web crawler used for data collection at scale. Powers datasets used by AI companies, market researchers, and enterprise data analytics platforms.
Purpose
Large-scale data collection for AI and analytics
User Agent String
Mozilla/5.0 (compatible; BrightBot/1.0; +https://brightdata.com)
How to Control in robots.txt
🚫 Block Brightbot
User-agent: Brightbot Disallow: /
✅ Allow Brightbot
User-agent: Brightbot Allow: /
Is Brightbot crawling your site?
Run a free scan to check if Bright Data's crawler is accessing your website.
Check if Brightbot is crawling YOUR site →