How to Block AI Bots on Clojure Ring: Complete 2026 Guide
Ring is Clojure's web middleware specification — the foundation beneath Compojure, Reitit, Luminus, and most Clojure web frameworks. A Ring middleware is a higher-order function: it accepts a handler and returns a new handler. To block AI bots, return a 403 response map without calling the inner handler — zero downstream execution.
Ring lowercases all header names
Ring normalizes HTTP header names to lowercase strings in the request map. Always use "user-agent" (lowercase) — not "User-Agent". Access it with (get-in request [:headers "user-agent"]). This applies to all headers: content-type, authorization, accept, etc.
Protection layers
1
robots.txt — wrap-resource "public" + wrap-content-type — outermost middleware, intercepts before bot-blocker
2
noai meta tag — In HTML response strings, Hiccup, or Selmer base template
3
X-Robots-Tag header — (update response :headers merge {"x-robots-tag" "noai, noimageai"}) on all non-blocked responses
4
Hard 403 — global wrap — (wrap app wrap-bot-blocker) — applies to every request through the app handler
5
Hard 403 — Reitit per-route — :middleware [wrap-bot-blocker] in route data — scoped to specific routes or groups
Step 1 — Shared bot list (src/my_app/ai_bots.clj)
A plain Clojure vector of lowercase patterns. some short-circuits on the first match — no need to check all patterns once one hits. str/lower-case normalises the UA before matching.
; src/my_app/ai_bots.clj — shared bot list
(ns my-app.ai-bots
(:require [clojure.string :as str]))
(def ai-bot-patterns
[;; OpenAI
"gptbot" "chatgpt-user" "oai-searchbot"
;; Anthropic
"claudebot" "claude-web"
;; Common Crawl
"ccbot"
;; Bytedance
"bytespider"
;; Meta
"meta-externalagent"
;; Perplexity
"perplexitybot"
;; Google AI
"google-extended" "googleother"
;; Cohere
"cohere-ai"
;; Amazon
"amazonbot"
;; Diffbot
"diffbot"
;; AI2
"ai2bot"
;; DeepSeek
"deepseekbot"
;; Mistral
"mistralai-user"
;; xAI
"xai-bot"
;; You.com
"youbot"
;; DuckDuckGo AI
"duckassistbot"])
(defn ai-bot?
"Returns true if the user-agent string matches a known AI training bot."
[user-agent]
(when (string? user-agent)
(let [ua (str/lower-case user-agent)]
(some #(str/includes? ua %) ai-bot-patterns))))Step 2 — Ring middleware (wrap-bot-blocker)
The pattern is idiomatic Ring: a function that takes a handler and returns a (fn [request] ...). The inner handler is a closure — it's captured in scope but never called when the request is blocked. The update :headers merge pattern preserves any existing headers the inner handler set.
; src/my_app/middleware.clj — Ring middleware
(ns my-app.middleware
(:require [clojure.string :as str]
[my-app.ai-bots :as bots]))
(defn wrap-bot-blocker
"Ring middleware that blocks AI training bots with a 403.
Usage:
(def app (-> my-handler
(wrap-bot-blocker)))
For AI bots: returns 403 immediately — inner handler never called.
For legitimate requests: calls (handler request) and adds X-Robots-Tag."
[handler]
(fn [request]
;; Ring normalizes all header names to lowercase strings.
;; "User-Agent" from the browser becomes "user-agent" in the request map.
(let [ua (or (get-in request [:headers "user-agent"]) "")]
(if (bots/ai-bot? ua)
;; Short-circuit — handler never runs
{:status 403
:headers {"content-type" "text/plain; charset=utf-8"
"x-robots-tag" "noai, noimageai"}
:body "Forbidden"}
;; Pass through — run the handler and add X-Robots-Tag
(-> (handler request)
(update :headers merge {"x-robots-tag" "noai, noimageai"}))))))Step 3 — Compojure integration
Compose middleware with the thread-first macro ->. Order matters: the outermost wrapper runs first on incoming requests. Place wrap-resource outermost so robots.txt is served before wrap-bot-blocker sees the request.
; src/my_app/core.clj — Compojure integration
(ns my-app.core
(:require [compojure.core :refer [defroutes GET POST]]
[compojure.route :as route]
[ring.adapter.jetty :as jetty]
[ring.middleware.defaults :refer [wrap-defaults site-defaults]]
[ring.middleware.resource :refer [wrap-resource]]
[ring.middleware.content-type :refer [wrap-content-type]]
[my-app.middleware :refer [wrap-bot-blocker]]))
(defroutes app-routes
;; Public routes — these still run through the middleware stack above,
;; but /robots.txt is intercepted by wrap-resource before reaching bot-blocker.
(GET "/health" [] {:status 200 :body "ok"})
;; Protected routes — wrap-bot-blocker is global, applies to all
(GET "/" []
{:status 200
:headers {"content-type" "text/html; charset=utf-8"}
:body "<html><head><meta name=\"robots\" content=\"noai, noimageai\"></head><body><h1>Welcome</h1></body></html>"})
(GET "/api/data" []
{:status 200
:headers {"content-type" "application/json"}
:body "{"data": "protected"}"})
(route/not-found {:status 404 :body "Not Found"}))
;; Middleware composition with ->
;; Execution order: wrap-resource → wrap-bot-blocker → wrap-defaults → routes
;; The first middleware in the chain handles the request first.
(def app
(-> app-routes
;; wrap-defaults FIRST (innermost) — applies defaults like cookie handling
(wrap-defaults site-defaults)
;; Bot blocker SECOND — applied after defaults, before static files
(wrap-bot-blocker)
;; wrap-resource LAST (outermost) — intercepts /robots.txt first, before bot-blocker
(wrap-resource "public")
(wrap-content-type)))
;; Start Jetty server
(defn -main [& _args]
(jetty/run-jetty app {:port 3000 :join? false}))Step 4 — Reitit: per-route middleware via route data
Reitit supports both global wrapping (same as Compojure) and per-route middleware via the :middleware key in route data. Per-route middleware only runs on matched routes — more granular than global wrapping.
; Reitit integration — middleware on specific routes or globally
(ns my-app.reitit-core
(:require [reitit.ring :as ring]
[reitit.ring.middleware.parameters :as parameters]
[ring.adapter.jetty :as jetty]
[ring.middleware.resource :refer [wrap-resource]]
[ring.middleware.content-type :refer [wrap-content-type]]
[my-app.middleware :refer [wrap-bot-blocker]]))
;; Reitit route data — middleware per-route or per-group
(def routes
[["/health"
{:get (fn [_] {:status 200 :body "ok"})}]
;; /api/* routes — bot blocker applied via route-level middleware
["/api"
{:middleware [wrap-bot-blocker]}
["/data"
{:get (fn [_] {:status 200
:headers {"content-type" "application/json"}
:body "{"data": "protected"}"})}]
["/users"
{:get (fn [_] {:status 200
:headers {"content-type" "application/json"}
:body "[]"})}]]
;; / — also protected by bot blocker at this route level
["/"
{:middleware [wrap-bot-blocker]
:get (fn [_] {:status 200 :body "Welcome"})}]])
(def app
(-> (ring/router routes)
(ring/ring-handler
;; Default handler for unmatched routes
(ring/create-default-handler
{:not-found (constantly {:status 404 :body "Not Found"})}))
;; Static files (robots.txt) — outermost, runs first
(wrap-resource "public")
(wrap-content-type)))
;; Alternative: global bot-blocker via ring-handler options
(def app-global
(-> (ring/router routes {:data {:middleware [parameters/parameters-middleware]}})
(ring/ring-handler
(ring/create-default-handler))
;; Global wrap — applies to every request regardless of route match
(wrap-bot-blocker)
(wrap-resource "public")
(wrap-content-type)))Step 5 — Server adapters (http-kit, Aleph, Jetty)
Your Ring handler works with any Ring-compatible server adapter. Swap Jetty for http-kit or Aleph by changing a single function call — the middleware stack is identical.
; http-kit server — same app, different adapter
(ns my-app.server
(:require [org.httpkit.server :as http-kit]
[my-app.core :refer [app]]))
;; The 'app' Ring handler works identically with any Ring-compatible server.
;; Switch from Jetty to http-kit by changing only the start call.
(defn start-server! []
(http-kit/run-server app {:port 3000}))
;; deps.edn — choose your server adapter:
;; Jetty: ring/ring-jetty-adapter {:mvn/version "1.12.1"}
;; http-kit: http-kit/http-kit {:mvn/version "2.8.0"}
;; Aleph: aleph/aleph {:mvn/version "0.7.1"}Step 6 — robots.txt
Place robots.txt in resources/public/. wrap-resource serves classpath resources — files in resources/ are automatically on the classpath in Leiningen and deps.edn projects.
; resources/public/robots.txt
; Place this file in your resources/public/ directory.
; ring.middleware.resource/wrap-resource serves it at GET /robots.txt.
; No code needed — just the file.
User-agent: *
Allow: /
# AI training bots — blocked
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: YouBot
Disallow: /
User-agent: AmazonBot
Disallow: /
User-agent: Diffbot
Disallow: /
; deps.edn — ring middleware dependencies:
; ring/ring-core {:mvn/version "1.12.1"}
; ring/ring-defaults {:mvn/version "0.5.0"}
; ring/ring-jetty-adapter {:mvn/version "1.12.1"}
; compojure/compojure {:mvn/version "1.7.1"} ; optional
; metosin/reitit {:mvn/version "0.7.2"} ; optionalStep 7 — noai meta tag in HTML responses
Three approaches: inline HTML string, Hiccup (HTML as Clojure data), or Selmer templates. The wrap-bot-blocker middleware adds the X-Robots-Tag header to all legitimate responses — so even if a page doesn't have the meta tag, the header still signals crawlers.
; noai meta tag in Ring HTML responses
(ns my-app.views)
;; Option A: Inline HTML string in a Ring response map
(defn html-response [title body-content]
{:status 200
:headers {"content-type" "text/html; charset=utf-8"
"x-robots-tag" "noai, noimageai"}
:body (str "<!DOCTYPE html><html><head>"
"<meta name=\"robots\" content=\"noai, noimageai\">"
"<title>" title "</title>"
"</head><body>" body-content "</body></html>")})
;; Option B: Hiccup (HTML as Clojure data structures)
;; deps.edn: hiccup/hiccup {:mvn/version "2.0.0-RC3"}
(require '[hiccup2.core :as h])
(defn page [title]
{:status 200
:headers {"content-type" "text/html; charset=utf-8"
"x-robots-tag" "noai, noimageai"}
:body (str (h/html
[:html
[:head
[:meta {:name "robots" :content "noai, noimageai"}]
[:title title]]
[:body
[:h1 "Welcome"]]]))})
;; Option C: Selmer template (Django-style)
;; Place <meta name="robots" content="noai, noimageai"> in base.html template.
;; The wrap-bot-blocker middleware adds X-Robots-Tag to the response headers.Ring vs Compojure vs Reitit vs Pedestal
| Feature | Ring (standalone) | Compojure | Reitit | Pedestal |
|---|---|---|---|---|
| Middleware model | Higher-order function: (defn wrap-x [handler] (fn [req] ...)) | Same Ring middleware — Compojure wraps its routes as Ring handlers | Ring middleware (global) or route data :middleware (per-route/group) | Interceptor chain with :enter/:leave/:error — bidirectional, not wrap |
| Short-circuit request | Return response map without calling (handler request) | Identical — same Ring middleware model | Identical — same Ring middleware model | :enter returns context map with :response set — bypasses later interceptors |
| Global vs scoped | (wrap app middleware) — global. No scope concept natively. | (wrap routes middleware) — global. context macro for path scoping. | Global via outer wrap, or :middleware in route data for per-group | Interceptors per-route in route table or shared in common-interceptors |
| UA header access | (get-in request [:headers "user-agent"]) — always lowercase | Identical — same request map structure | Identical — same request map structure | (get-in ctx [:request :headers "user-agent"]) |
| Hard 403 | {:status 403, :headers {...}, :body "Forbidden"} | Identical map format | Identical map format | (assoc context :response {:status 403, :body "Forbidden"}) |
| robots.txt | wrap-resource "public" + wrap-content-type | Same wrap-resource middleware | Same wrap-resource or reitit.ring/create-resource-handler | Pedestal route for /robots.txt or servlet context resource serving |
| HTTP server | Jetty (ring-jetty-adapter), http-kit, or Aleph | Same — Compojure is router-only, not a server | Same — Reitit is router-only, not a server | Jetty, Tomcat, or Immutant (via pedestal.service) |
Summary
- Higher-order function pattern —
(defn wrap-x [handler] (fn [req] ...)). Return the map directly for AI bots; call(handler req)for legitimate requests. - Headers are always lowercase —
"user-agent", never"User-Agent". - wrap-resource outermost — robots.txt must be reachable before the bot-blocker runs. Outermost = first to handle the request.
- update :headers merge — adds X-Robots-Tag without clobbering headers the inner handler set.
- Server-agnostic — same Ring handler works on Jetty, http-kit, and Aleph. One-line server swap.
Is your site protected from AI bots?
Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.