Skip to content
Guides/Chi (Go)

How to Block AI Bots on Chi (Go): Complete 2026 Guide

Chi is a lightweight, idiomatic Go HTTP router built directly on net/http. Unlike Gin and Echo, Chi introduces zero custom types for middleware — it uses the standard func(http.Handler) http.Handler signature. Block with http.Error(w, "Forbidden", 403) (Layer 4), or set w.Header().Set() before next.ServeHTTP() for X-Robots-Tag (Layer 3).

Pure net/http — zero custom types

Chi middleware is func(http.Handler) http.Handler — the same signature as net/http middleware. Inside, you have http.ResponseWriter and *http.Request — no *gin.Context, no echo.Context, no *fiber.Ctx. Any middleware you write for Chi works with the standard library's http.ServeMux and vice versa. Gin uses a chain model with c.Next()/c.Abort(). Echo wraps echo.HandlerFunc. Chi wraps http.Handler — the standard.

Protection layers

1
robots.txt — http.FileServer or http.ServeFile at /robots.txt — registered as a route
2
noai meta tag — html/template layout — {{ .Robots }} variable with default
3
X-Robots-Tag header — w.Header().Set("X-Robots-Tag", "noai, noimageai") before next.ServeHTTP()
4
Hard 403 block — http.Error(w, "Forbidden", http.StatusForbidden) — do not call next

Layer 1: robots.txt

Create a static/ directory at your project root (alongside main.go and go.mod) and place robots.txt there. Serve it as a route or exempt it in the middleware.

# static/robots.txt

User-agent: *
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Diffbot
User-agent: cohere-ai
User-agent: FacebookBot
User-agent: omgili
User-agent: omgilibot
User-agent: Amazonbot
User-agent: DeepSeekBot
User-agent: MistralBot
User-agent: xAI-Bot
User-agent: AI2Bot
Disallow: /

Serve via Chi route:

r := chi.NewRouter()

// Serve robots.txt as a dedicated route
r.Get("/robots.txt", func(w http.ResponseWriter, r *http.Request) {
    http.ServeFile(w, r, "static/robots.txt")
})

// Then register bot blocker
r.Use(AiBotBlocker)

Alternatively, generate dynamically:

r.Get("/robots.txt", func(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "text/plain")
    fmt.Fprint(w, `User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /`)
})

Layer 2: noai meta tag

If your Chi app renders HTML via html/template, add the noai meta tag to your base layout:

Go html/template layout (templates/layout.html)

<!-- templates/layout.html -->
{{- if .Robots }}
<meta name="robots" content="{{ .Robots }}">
{{- else }}
<meta name="robots" content="noai, noimageai">
{{- end }}

Render with override in Chi route handler

r.Get("/public-page", func(w http.ResponseWriter, r *http.Request) {
    tmpl.ExecuteTemplate(w, "layout.html", map[string]any{
        "Robots": "index, follow",
    })
})

// Routes that omit "Robots" get the default noai value

If your Chi app is a JSON API with a separate SPA frontend (React, Vue, Svelte), add the noai meta tag in the frontend's base layout instead.

Layers 3 & 4: net/http middleware

Chi middleware is func(next http.Handler) http.Handler — identical to the standard library. No custom types, no learning curve.

middleware/aibot.go

package middleware

import (
	"net/http"
	"strings"
)

var aiBotPatterns = []string{
	"gptbot", "chatgpt-user", "oai-searchbot",
	"claudebot", "anthropic-ai", "claude-web",
	"google-extended", "ccbot", "bytespider",
	"applebot-extended", "perplexitybot", "diffbot",
	"cohere-ai", "facebookbot", "meta-externalagent",
	"omgili", "omgilibot", "amazonbot",
	"deepseekbot", "mistralbot", "xai-bot", "ai2-bot",
}

var exemptPaths = map[string]bool{
	"/robots.txt":  true,
	"/sitemap.xml": true,
	"/favicon.ico": true,
}

// AiBotBlocker returns an http.Handler that blocks known AI training crawlers.
// This is standard net/http middleware — works with Chi, http.ServeMux, and any
// router that accepts func(http.Handler) http.Handler.
func AiBotBlocker(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// Always pass through exempt paths
		if exemptPaths[r.URL.Path] {
			next.ServeHTTP(w, r)
			return
		}

		ua := strings.ToLower(r.Header.Get("User-Agent"))

		for _, pattern := range aiBotPatterns {
			if strings.Contains(ua, pattern) {
				// Layer 4: hard block — 403 Forbidden, do NOT call next
				http.Error(w, "Forbidden", http.StatusForbidden)
				return
			}
		}

		// Layer 3: set X-Robots-Tag BEFORE calling next — headers in the map
		// are sent when the handler calls WriteHeader() or Write()
		w.Header().Set("X-Robots-Tag", "noai, noimageai")
		next.ServeHTTP(w, r)
	})
}

Key points

  • Blocking: http.Error(w, "Forbidden", http.StatusForbidden) writes the status code and body, then return prevents next.ServeHTTP() from running. The route handler never executes.
  • X-Robots-Tag before next: w.Header().Set() modifies the header map. When the inner handler calls w.WriteHeader() or w.Write(), all headers already in the map are sent to the client. Setting headers after next.ServeHTTP() is unreliable — the handler may have already flushed headers.
  • Reading request headers: r.Header.Get("User-Agent") — standard *http.Request.
  • Writing response headers: w.Header().Set(key, value) — standard http.ResponseWriter.
  • No custom types: this middleware compiles and runs with http.ServeMux, chi.Router, or any Go HTTP router that accepts the standard middleware signature. Zero vendor lock-in.

Why set headers before next, not after?

In Go's net/http, w.Header() returns the header map. When WriteHeader() is called (explicitly, or implicitly on first Write()), the map is frozen and sent to the client. After that point, w.Header().Set() has no effect. Because you don't control when the inner handler writes, set response headers before next.ServeHTTP(w, r). This differs from Echo and Gin, where setting headers after next(c) /c.Next() often works because their response writers buffer headers differently.

Registering the middleware

package main

import (
	"fmt"
	"log"
	"net/http"

	"github.com/go-chi/chi/v5"
	chimiddleware "github.com/go-chi/chi/v5/middleware"
	"yourapp/middleware"
)

func main() {
	r := chi.NewRouter()

	// Chi's built-in middleware
	r.Use(chimiddleware.Logger)
	r.Use(chimiddleware.Recoverer)

	// Bot blocker — runs on every route after Logger/Recoverer
	r.Use(middleware.AiBotBlocker)

	// robots.txt route — exempt in middleware via exemptPaths
	r.Get("/robots.txt", func(w http.ResponseWriter, r *http.Request) {
		http.ServeFile(w, r, "static/robots.txt")
	})

	// Your routes
	r.Get("/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprint(w, "Hello, World!")
	})
	r.Get("/api/data", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "application/json")
		fmt.Fprint(w, `{"status":"ok"}`)
	})

	log.Fatal(http.ListenAndServe(":3000", r))
}

Chi runs middleware in FIFO order — the first r.Use() call runs first (outermost wrapper). Register the bot blocker before auth, CORS, and body parsing middlewares. Logger and Recoverer are typically first so that blocked requests are still logged and panics are still recovered.

Route-group blocking

Chi provides two grouping mechanisms: r.Route() for path-prefixed sub-routers and r.Group() for inline groups on the same path.

// Public routes — no bot blocking
r.Get("/", homeHandler)
r.Get("/about", aboutHandler)

// API routes — bot blocker scoped to /api/*
r.Route("/api", func(api chi.Router) {
    api.Use(middleware.AiBotBlocker)
    api.Get("/products", productsHandler)
    api.Get("/users", usersHandler)
    api.Post("/orders", ordersHandler)
})

r.Route("/api", fn) creates a sub-router mounted at /api. api.Use() adds middleware only for that sub-router. Routes outside /api are unaffected.

Inline groups with r.Group()

r.Group() creates a middleware group without a path prefix — useful when you want different middleware for different routes at the same path level:

// Public group — no bot blocking
r.Group(func(r chi.Router) {
    r.Get("/", homeHandler)
    r.Get("/about", aboutHandler)
})

// Protected group — bot blocking applied
r.Group(func(r chi.Router) {
    r.Use(middleware.AiBotBlocker)
    r.Get("/dashboard", dashboardHandler)
    r.Get("/settings", settingsHandler)
})

Comparison: Chi vs Gin vs Echo vs net/http

Chi (pure net/http)

// Middleware: func(http.Handler) http.Handler
func AiBotBlocker(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if isBot(r) {
            http.Error(w, "Forbidden", 403)
            return
        }
        w.Header().Set("X-Robots-Tag", "noai, noimageai")
        next.ServeHTTP(w, r)
    })
}
// Register: r.Use(AiBotBlocker)

Gin (chain model)

// Middleware: func(*gin.Context) — via factory
func AiBotBlocker() gin.HandlerFunc {
    return func(c *gin.Context) {
        if isBot(c.Request) {
            c.AbortWithStatus(403)
            return
        }
        c.Next()
        c.Header("X-Robots-Tag", "noai, noimageai")
    }
}
// Register: r.Use(AiBotBlocker())

Echo (wrapper pattern)

// Middleware: func(echo.HandlerFunc) echo.HandlerFunc
func AiBotBlocker(next echo.HandlerFunc) echo.HandlerFunc {
    return func(c echo.Context) error {
        if isBot(c.Request()) {
            return echo.NewHTTPError(403, "Forbidden")
        }
        err := next(c)
        c.Response().Header().Set("X-Robots-Tag", "noai, noimageai")
        return err
    }
}
// Register: e.Use(AiBotBlocker)

net/http (standard library)

// Middleware: func(http.Handler) http.Handler
// Identical to Chi — Chi uses net/http middleware directly
func AiBotBlocker(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if isBot(r) {
            http.Error(w, "Forbidden", 403)
            return
        }
        w.Header().Set("X-Robots-Tag", "noai, noimageai")
        next.ServeHTTP(w, r)
    })
}
// Register: http.ListenAndServe(addr, AiBotBlocker(mux))

Notice that Chi and net/http are identical. Chi adds URL parameters, sub-routers, and r.Use() convenience — but middleware is pure net/http. Any middleware you write for one works with the other.

Chi-specific patterns

URL parameter access

If your middleware needs to read Chi URL parameters (e.g., to exempt specific resource IDs):

import "github.com/go-chi/chi/v5"

// Inside a middleware or handler:
productID := chi.URLParam(r, "productID")

// Route definition:
r.Route("/products/{productID}", func(r chi.Router) {
    r.Get("/", getProduct)
})

Chi's built-in middleware

Chi ships with a chi/middleware package. Common stack with bot blocker:

import chimiddleware "github.com/go-chi/chi/v5/middleware"

r := chi.NewRouter()

// 1. RealIP — extracts client IP from X-Forwarded-For / X-Real-IP
r.Use(chimiddleware.RealIP)

// 2. Logger — logs method, path, status, latency
r.Use(chimiddleware.Logger)

// 3. Recoverer — catches panics, returns 500 instead of crashing
r.Use(chimiddleware.Recoverer)

// 4. Bot blocker — blocks AI crawlers
r.Use(middleware.AiBotBlocker)

// Logger runs before bot blocker — blocked requests are logged.
// Move bot blocker before Logger to skip logging blocked bots.

Mounting sub-routers

Chi supports mounting entirely separate routers, each with their own middleware stack:

// Public router — no bot blocking
publicRouter := chi.NewRouter()
publicRouter.Get("/", homeHandler)
publicRouter.Get("/about", aboutHandler)

// API router — bot blocking + auth
apiRouter := chi.NewRouter()
apiRouter.Use(middleware.AiBotBlocker)
apiRouter.Use(authMiddleware)
apiRouter.Get("/products", productsHandler)

// Mount both on the main router
r := chi.NewRouter()
r.Use(chimiddleware.Logger)
r.Mount("/", publicRouter)
r.Mount("/api", apiRouter)

Verify it works

# Test blocked bot — should return 403
curl -A "Mozilla/5.0 (compatible; GPTBot/1.0)" http://localhost:3000/api/data

# Test legitimate request — should return 200 with X-Robots-Tag
curl -v -A "Mozilla/5.0 (Macintosh)" http://localhost:3000/api/data 2>&1 | grep -i x-robots-tag

# Test robots.txt — should return 200 even with bot UA
curl -A "Mozilla/5.0 (compatible; GPTBot/1.0)" http://localhost:3000/robots.txt

Related Go guides

→ net/http (stdlib)→ Gin→ Echo→ Fiber (fasthttp)

Is your site protected from AI bots?

Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.