About
A crawler specializing in forums, discussions, and user-generated content. Powers Webz.io's data feeds which are commonly used for AI training and sentiment analysis.
Purpose
Forum and discussion content crawling for AI datasets
User Agent String
Mozilla/5.0 (compatible; omgili/0.5; +http://omgili.com)
How to Control in robots.txt
🚫 Block Omgili
User-agent: Omgili Disallow: /
✅ Allow Omgili
User-agent: Omgili Allow: /
Complete Guide: How to Block Omgili
Server-level blocking, nginx configs, Cloudflare rules, Next.js middleware, and more →
Is Omgili crawling your site?
Enter your URL below — scan takes under 5 seconds.
Free · No signup · Instant results