How to Block AI Bots on Bottle (Python): Complete 2026 Guide
Bottle is Python's single-file micro-framework — zero dependencies, one file, WSGI. Unlike Flask (return a response to block) and Sanic (return an HTTPResponse from on_request), Bottle uses an abort-based hook system: call abort(403) in a before_request hook to block. Hooks cannot return responses — they raise exceptions to interrupt flow.
Abort-based blocking — not return-based
Bottle's before_request hook blocks by calling abort(403), which raises an HTTPError exception. This is the same pattern as Falcon (raise HTTPForbidden()). Unlike Flask, you cannot return a response from a Bottle hook to short-circuit — returning has no effect.
Protection layers
Layer 1: robots.txt
Bottle doesn't auto-serve static files like Flask. Use a dedicated route with static_file() and exempt it in your hook:
# static/robots.txt User-agent: * Allow: / User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: CCBot User-agent: cohere-ai User-agent: Bytespider User-agent: Amazonbot User-agent: PerplexityBot User-agent: YouBot User-agent: Diffbot User-agent: DeepSeekBot User-agent: MistralBot User-agent: xAI-Bot User-agent: AI2Bot Disallow: /
# app.py — serve robots.txt
from bottle import Bottle, static_file
app = Bottle()
@app.route('/robots.txt')
def robots():
return static_file('robots.txt', root='./static/')
@app.route('/sitemap.xml')
def sitemap():
return static_file('sitemap.xml', root='./static/')Bottle static routes are regular routes —
before_request hooks run before them. You must explicitly exempt /robots.txt in your blocker. This differs from Flask (auto-serves /static/ before before_request) and Sanic (app.static() bypasses on_request).Layers 2, 3 & 4: hook-based blocking
Bottle's hook system has two relevant execution points: before_request (runs before route handler) and after_request (runs after):
# app.py
from bottle import Bottle, request, response, abort
app = Bottle()
AI_BOTS = [
'gptbot', 'chatgpt-user', 'claudebot', 'anthropic-ai',
'ccbot', 'cohere-ai', 'bytespider', 'amazonbot',
'applebot-extended', 'perplexitybot', 'youbot', 'diffbot',
'google-extended', 'deepseekbot', 'mistralbot', 'xai-bot',
'ai2bot', 'oai-searchbot', 'duckassistbot',
]
EXEMPT_PATHS = {'/robots.txt', '/sitemap.xml', '/favicon.ico'}
@app.hook('before_request')
def block_ai_bots():
"""Layer 2 + 4: Set noai meta context and block AI bots."""
# Layer 2: Store noai meta for templates
request.environ['robots'] = 'noai, noimageai'
# Exempt paths — robots.txt must always be accessible
if request.path in EXEMPT_PATHS:
return
# Layer 4: Hard block for AI bots
ua = request.headers.get('User-Agent', '').lower()
if any(bot in ua for bot in AI_BOTS):
abort(403, 'Forbidden: AI crawlers are not permitted.')
@app.hook('after_request')
def inject_robots_tag():
"""Layer 3: Add X-Robots-Tag to every response."""
response.set_header('X-Robots-Tag', 'noai, noimageai')abort(403) raises an HTTPError exception. The route handler never executes. Any value returned from a before_request hook is silently ignored — you must use abort() or raise an exception to stop the request.Layer 2: noai meta tag
Bottle doesn't have Flask's g or Sanic's request.ctx. Use request.environ (the WSGI environ dict) to store per-request data:
# In before_request hook (already set above):
request.environ['robots'] = 'noai, noimageai'
# In SimpleTemplate (Bottle's built-in):
# views/base.tpl
# <meta name="robots" content="{{request.environ.get('robots', 'noai, noimageai')}}">
# In Jinja2 template (if using bottle-jinja2):
# <meta name="robots" content="{{ request.environ.get('robots', 'noai, noimageai') }}">
# Route handler — override per-page:
@app.route('/public-page')
def public_page():
request.environ['robots'] = 'index, follow' # Override
return template('page', request=request)Bottle:
request.environ['robots'] — WSGI environ dict.Flask:
g.robots — thread-local proxy.Sanic:
request.ctx.robots — SimpleNamespace.aiohttp:
request['robots'] — MutableMapping.Plugin system (per-route control)
Bottle hooks are always global. For per-route control (opt-out specific routes), use Bottle's plugin system. Plugins implement apply(callback, route) and can inspect route config:
from bottle import Bottle, request, abort
app = Bottle()
AI_BOTS = ['gptbot', 'claudebot', 'ccbot', 'anthropic-ai', ...]
class AiBotBlockerPlugin:
"""Bottle plugin — wraps routes, respects skip_bot_check config."""
name = 'ai_bot_blocker'
api = 2
def apply(self, callback, route):
# Check if route opts out of bot blocking
skip = route.config.get('skip_bot_check', False)
if skip:
return callback # Return unwrapped handler
def wrapper(*args, **kwargs):
ua = request.headers.get('User-Agent', '').lower()
if any(bot in ua for bot in AI_BOTS):
abort(403, 'Forbidden')
return callback(*args, **kwargs)
return wrapper
# Install plugin globally
app.install(AiBotBlockerPlugin())
# This route IS protected (default):
@app.route('/api/data')
def api_data():
return {'results': [1, 2, 3]}
# This route opts OUT of bot blocking:
@app.route('/health', skip_bot_check=True)
def health():
return 'OK'
# This route also opts out:
@app.route('/robots.txt', skip_bot_check=True)
def robots():
return static_file('robots.txt', root='./static/')Plugin api = 2 is required for Bottle 0.12+. Route config keys (like skip_bot_check) are passed as keyword arguments to @app.route().
Sub-app mounting for path scoping
For path-prefixed isolation (like aiohttp sub-applications or Express sub-routers), use Bottle's mount():
from bottle import Bottle, request, abort
# Main app — no bot blocking
main = Bottle()
@main.route('/')
def index():
return 'Hello!' # Bots can access this
# API sub-app — has its own bot blocking hook
api = Bottle()
@api.hook('before_request')
def block_bots_api():
ua = request.headers.get('User-Agent', '').lower()
if any(bot in ua for bot in AI_BOTS):
abort(403, 'Forbidden')
@api.route('/data')
def api_data():
return {'results': [1, 2, 3]}
# Mount API at /api/
main.mount('/api', api)
# /api/data → runs api's before_request hook (bot blocking)
# / → no bot blockingMounted sub-apps have their own independent hook registrations. The parent app's hooks do not run for mounted routes — Bottle delegates entirely to the sub-app.
Bottle vs Flask vs Falcon — blocking comparison
Bottle — abort() raises HTTPError
# bottle hook (abort-based)
@app.hook('before_request')
def block_bots():
ua = request.headers.get('User-Agent', '').lower()
if any(b in ua for b in AI_BOTS):
abort(403, 'Forbidden') # raises HTTPErrorFlask — return Response to block
# flask before_request (return-based)
@app.before_request
def block_bots():
ua = request.headers.get('User-Agent', '').lower()
if any(b in ua for b in AI_BOTS):
return Response('Forbidden', 403) # return stopsFalcon — raise exception
# falcon middleware (raise-based)
def process_request(self, req, resp):
ua = req.get_header('User-Agent') or ''
if any(b in ua.lower() for b in AI_BOTS):
raise falcon.HTTPForbidden() # raise stopsDjango — return HttpResponse
# django middleware (return-based)
def process_request(self, request):
ua = request.META.get('HTTP_USER_AGENT', '').lower()
if any(b in ua for b in AI_BOTS):
return HttpResponseForbidden('Forbidden')Bottle and Falcon both use exception/abort to block. Flask and Django use return-based blocking. In Bottle, returning from a hook has no effect — you must abort() or raise.
after_request vs after_request on error
Bottle's after_request hook runs after every response — including error responses from abort(). This means your X-Robots-Tag will be set even on 403 blocked responses:
@app.hook('after_request')
def inject_robots_tag():
# Runs on ALL responses — 200, 403, 404, 500, etc.
response.set_header('X-Robots-Tag', 'noai, noimageai')
# To skip on blocked responses:
# if response.status_code != 403:
# response.set_header('X-Robots-Tag', 'noai, noimageai')Unlike Flask's
after_request (skipped on unhandled exceptions unless teardown_request is used), Bottle's after_request runs on all responses including errors. The response object is always populated.Testing
Bottle includes webtest integration. Use the TestApp wrapper:
import pytest
from webtest import TestApp
from app import app # Your Bottle app
@pytest.fixture
def client():
return TestApp(app)
def test_blocks_ai_bot(client):
resp = client.get(
'/api/data',
headers={'User-Agent': 'GPTBot/1.0'},
expect_errors=True,
)
assert resp.status_int == 403
def test_allows_browser(client):
resp = client.get(
'/api/data',
headers={'User-Agent': 'Mozilla/5.0 (compatible)'},
)
assert resp.status_int == 200
assert resp.headers.get('X-Robots-Tag') == 'noai, noimageai'
def test_robots_txt_accessible_to_bots(client):
resp = client.get(
'/robots.txt',
headers={'User-Agent': 'GPTBot/1.0'},
)
assert resp.status_int == 200 # Exempt path — not blocked
def test_skip_bot_check_route(client):
resp = client.get(
'/health',
headers={'User-Agent': 'ClaudeBot/1.0'},
)
assert resp.status_int == 200 # skip_bot_check=TrueAI bot User-Agent strings (2026)
Bottle uses standard WSGI — User-Agent is in request.headers.get('User-Agent', '').lower(). The HeaderDict is case-insensitive.
Is your site protected from AI bots?
Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.