How to Block AI Bots in Python Quart

1. Bot detection utility

A plain Python module with no async — detection is CPU-bound, so there is no benefit to making it async. str.find() performs literal substring matching with no regex engine. Applied to the lowercased UA string once before iterating.

# bot_utils.py — shared bot detection, no external dependencies
from __future__ import annotations

# All lowercase — matched against ua.lower()
_AI_BOT_PATTERNS: tuple[str, ...] = (
    "gptbot",
    "chatgpt-user",
    "claudebot",
    "anthropic-ai",
    "ccbot",
    "google-extended",
    "cohere-ai",
    "meta-externalagent",
    "bytespider",
    "omgili",
    "diffbot",
    "imagesiftbot",
    "magpie-crawler",
    "amazonbot",
    "dataprovider",
    "netcraft",
)


def is_ai_bot(ua: str) -> bool:
    """Return True if the User-Agent string matches a known AI crawler."""
    if not ua:
        return False
    lower = ua.lower()
    # str.find() — literal substring, no regex engine, no backtracking
    return any(lower.find(p) != -1 for p in _AI_BOT_PATTERNS)

2. before_request + after_request hooks

Register an async def function with @app.before_request. Return None to pass through; return a Response (or call abort()) to block. The after_request hook injects X-Robots-Tag on all responses that were not aborted.

# app.py — Quart application with async before_request hook
from __future__ import annotations

from quart import Quart, abort, request, Response

from bot_utils import is_ai_bot

app = Quart(__name__)


# ── AI-bot blocker — async before_request hook ────────────────────────────────
# Fires before every request, including static file requests.
# Any non-None return value short-circuits the view function.
@app.before_request
async def block_ai_bots() -> Response | None:
    # Path guard: let robots.txt through so crawlers can read their rules.
    # Remove this guard if you serve robots.txt via Nginx upstream of Quart.
    if request.path == "/robots.txt":
        return None  # pass through

    ua: str = request.headers.get("User-Agent", "")

    if is_ai_bot(ua):
        # abort() raises an HTTPException — Quart catches it and renders
        # the error response. No explicit return needed after abort().
        abort(403)

    return None  # pass through — let the view function run


# ── X-Robots-Tag on all passing responses ─────────────────────────────────────
# after_request fires for responses that were NOT aborted.
# Aborted requests are handled by the error handler, not after_request.
# To add headers to error responses too, use an @app.errorhandler.
@app.after_request
async def add_robots_header(response: Response) -> Response:
    response.headers["X-Robots-Tag"] = "noai, noimageai"
    return response


# ── Routes ────────────────────────────────────────────────────────────────────
@app.route("/")
async def index() -> str:
    return "Hello"


@app.route("/api/data")
async def api_data() -> Response:
    return app.response_class(
        response='{"data":"value"}',
        status=200,
        mimetype="application/json",
    )


if __name__ == "__main__":
    # Development server — use Hypercorn for production
    app.run()

3. Return a Response directly (make_response variant)

Use await make_response() when you need a custom response body or specific headers on the blocked response. Note: make_response() is a coroutine in Quart — it must be awaited. This is the most common gotcha when porting Flask code to Quart.

# Alternative: return a Response directly instead of using abort()
# Use this pattern when you need custom response body or headers on the block.

from quart import make_response

@app.before_request
async def block_ai_bots() -> Response | None:
    if request.path == "/robots.txt":
        return None

    ua: str = request.headers.get("User-Agent", "")

    if is_ai_bot(ua):
        # make_response() in Quart is a coroutine — must be awaited.
        # This is a key difference from Flask where make_response() is synchronous.
        response = await make_response("Forbidden", 403)
        response.headers["X-Robots-Tag"] = "noai, noimageai"
        response.headers["Content-Type"] = "text/plain"
        return response  # non-None return skips the view function

    return None

4. Error handler — X-Robots-Tag on abort(403) responses

after_request does not fire for requests that were aborted — Quart routes aborted requests to the error handler pipeline instead. Register an @app.errorhandler(403) to add X-Robots-Tag to blocked responses too. With both hooks in place, every response gets the header with no duplication.

# Add X-Robots-Tag to 403 error responses as well
# after_request does NOT fire for aborted requests — use an error handler.

from quart import jsonify

@app.errorhandler(403)
async def forbidden(error: Exception) -> tuple[Response, int]:
    response = await make_response("Forbidden", 403)
    response.headers["X-Robots-Tag"] = "noai, noimageai"
    response.headers["Content-Type"] = "text/plain"
    return response, 403


# With both after_request and errorhandler(403), all responses get the header:
# - Passing requests:  after_request adds X-Robots-Tag
# - Blocked requests:  errorhandler(403) adds X-Robots-Tag
# There is no duplication because after_request and errorhandler are disjoint.

5. Blueprint scoping — protect specific routes only

@blueprint.before_request registers a hook that fires only for routes on that blueprint. Use this when some endpoints (health checks, webhooks, internal APIs) should bypass the bot filter. The filter on the main app continues to apply to routes registered directly on the app.

# Blueprint scoping — apply bot blocker to specific blueprints only
# Useful when some endpoints (health checks, webhooks) should bypass the filter.

from quart import Blueprint

api_bp = Blueprint("api", __name__, url_prefix="/api")


@api_bp.before_request
async def block_bots_on_api() -> Response | None:
    """Applies only to routes registered on api_bp."""
    ua: str = request.headers.get("User-Agent", "")
    if is_ai_bot(ua):
        abort(403)
    return None


@api_bp.route("/data")
async def api_data() -> dict[str, str]:
    return {"data": "value"}


# Health check endpoint on the main app — bypasses the blueprint filter
@app.route("/health")
async def health() -> dict[str, str]:
    return {"status": "ok"}


app.register_blueprint(api_bp)

6. Async operations inside hooks (Redis rate limiting)

The defining advantage of Quart over Flask: you can await async I/O inside before_request hooks without blocking the event loop or using thread pools. This enables async Redis rate limiting, async database lookups, and async allow-list checks — all patterns that require significant workarounds in synchronous Flask.

# Async bot check with database or Redis rate limiting
# This is the key advantage of Quart over Flask — await inside hooks.

import aioredis  # or any async Redis/DB client

@app.before_request
async def block_ai_bots_with_ratelimit() -> Response | None:
    if request.path == "/robots.txt":
        return None

    ua: str = request.headers.get("User-Agent", "")

    if is_ai_bot(ua):
        # Async Redis lookup — impossible in synchronous Flask
        redis = await aioredis.create_redis("redis://localhost")
        ip = request.remote_addr or "unknown"
        await redis.incr(f"blocked:{ip}")
        redis.close()
        await redis.wait_closed()
        abort(403)

    return None

7. Production deployment — Hypercorn

Quart runs on any ASGI server. Hypercorn is the reference server maintained by the Quart author. Uvicorn is also fully compatible. Configure the event loop backend — asyncio (default) or trio. Trio provides structured concurrency and better cancellation semantics.

# hypercorn.toml — production ASGI server configuration
bind = ["0.0.0.0:8080"]
workers = 4
worker_class = "asyncio"   # or "trio" for Trio backend

# Access logging
accesslog = "-"            # stdout
errorlog = "-"             # stderr

# TLS (optional)
# certfile = "/etc/ssl/certs/app.crt"
# keyfile = "/etc/ssl/private/app.key"

# Run:
# hypercorn app:app --config hypercorn.toml

8. robots.txt

Quart serves the static/ directory at /static/ by default — not at /. To expose robots.txt at /robots.txt, add an explicit route or serve it from Nginx upstream of Quart. When served by Quart, the before_request path guard lets it through; when served by Nginx, the hook never fires for it.

# static/robots.txt (Quart serves static/ at /static/ by default)
# Or place at the root and add a route — see note below.

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# ── Serving robots.txt at / in Quart ─────────────────────────────────────────
# Quart serves static/ at /static/ by default — not at /.
# To serve robots.txt at /robots.txt, add a route:

# @app.route("/robots.txt")
# async def robots():
#     return await app.send_static_file("robots.txt")
#
# Or serve it from Nginx upstream of Quart:
# location = /robots.txt {
#     root /var/www/html;
#     add_header X-Robots-Tag "noai, noimageai";
# }

Key points

• make_response() is a coroutine in Quart: Unlike Flask where make_response() is synchronous, Quart's version must be awaited: response = await make_response("Forbidden", 403). Forgetting await returns a coroutine object, not a response — the most common Flask-to-Quart migration bug.
• before_request return value controls routing: Return None to continue to the view function; return any Response object to use it as the response. abort(403) raises an HTTPException that Quart catches — it also skips the view function and goes to the error handler.
• after_request does not fire for aborted requests: Quart routes aborted requests through the error handler pipeline, not through after_request. Register an @app.errorhandler(403) to add headers to blocked responses.
• Blueprint hooks are scoped: @blueprint.before_request fires only for routes on that blueprint. App-level @app.before_request fires for all routes including blueprints. They do not conflict — both run in sequence for blueprint routes.
• await inside hooks — the Quart advantage: Quart hooks are native coroutines running on the event loop. You can await any async I/O — Redis, PostgreSQL (asyncpg), HTTP clients (aiohttp, httpx) — without blocking other requests. Flask requires asyncio.run() workarounds or thread pools for the same pattern.
• Static file routing: Quart's built-in static handler serves static/ at /static/ — requests for these files go through before_request. To serve robots.txt at /robots.txt, add an explicit route or configure Nginx to serve it before requests reach Quart.

Framework comparison — Python async web frameworks

Quart and Flask share identical hook API surface — the only differences are async def and await make_response(). This makes Quart the lowest-friction migration path for Flask applications that need async I/O. FastAPI and Starlette use a different pattern: middleware functions with a call_next callable rather than before/after hooks.

Dependencies

pip install quart
pip install hypercorn          # ASGI server (recommended)
pip install uvicorn            # alternative ASGI server
pip install aioredis           # async Redis (optional, for rate limiting)

# Run development server
python app.py                  # uses Quart's built-in dev server

# Run with Hypercorn (production)
hypercorn app:app --bind 0.0.0.0:8080 --workers 4

# Run with Uvicorn
uvicorn app:app --host 0.0.0.0 --port 8080 --workers 4

# Trio backend (alternative event loop)
pip install trio
hypercorn app:app --worker-class trio