How to Block AI Bots in Go Buffalo

1. Bot pattern list

Define patterns in the actions package so they are accessible from both middleware and route handlers. strings.Contains with strings.ToLower is all that's needed — no regex, no external dependencies.

// actions/ai_bots.go
package actions

import "strings"

var aiBotPatterns = []string{
	"gptbot",
	"chatgpt-user",
	"claudebot",
	"anthropic-ai",
	"ccbot",
	"google-extended",
	"cohere-ai",
	"meta-externalagent",
	"bytespider",
	"omgili",
	"diffbot",
	"imagesiftbot",
	"magpie-crawler",
	"amazonbot",
	"dataprovider",
	"netcraft",
}

func isAiBot(userAgent string) bool {
	ua := strings.ToLower(userAgent)
	for _, pattern := range aiBotPatterns {
		if strings.Contains(ua, pattern) {
			return true
		}
	}
	return false
}

2. BotBlocker middleware

Buffalo middleware follows the func(next buffalo.Handler) buffalo.Handler pattern, where buffalo.Handler is func(c buffalo.Context) error. Set X-Robots-Tag at the top of the function — before any branch — so it appears on both blocked and passing responses without duplication.

// actions/middleware.go
package actions

import (
	"net/http"

	"github.com/gobuffalo/buffalo"
)

// BotBlocker sets X-Robots-Tag on every response and returns 403
// for known AI crawlers. Set the header BEFORE calling next(c) or
// returning so it appears on both blocked and passing responses.
func BotBlocker(next buffalo.Handler) buffalo.Handler {
	return func(c buffalo.Context) error {
		// Set header first — applies to every response path below
		c.Response().Header().Set("X-Robots-Tag", "noai, noimageai")

		// Allow robots.txt regardless of User-Agent
		if c.Request().URL.Path == "/robots.txt" {
			return next(c)
		}

		ua := c.Request().Header.Get("User-Agent")
		if isAiBot(ua) {
			return c.Render(http.StatusForbidden, r.String("Forbidden"))
		}

		return next(c)
	}
}

3. Register with app.Use()

Add app.Use(BotBlocker) in the App() function in app.go. Middleware runs in registration order. Place BotBlocker after Buffalo's built-in middleware (RequestID, ParameterLogger) but before your routes.

// app.go — register middleware with app.Use()
package actions

import (
	"github.com/gobuffalo/buffalo"
	"github.com/gobuffalo/buffalo/middleware"
	"github.com/gobuffalo/buffalo/middleware/csrf"
)

var app *buffalo.App

func App() *buffalo.App {
	if app == nil {
		app = buffalo.New(buffalo.Options{
			Env:         ENV,
			SessionName: "_myapp_session",
		})

		// Built-in middleware
		app.Use(middleware.RequestID)
		app.Use(middleware.ParameterLogger)
		app.Use(csrf.New)

		// AI bot blocker — runs before all routes
		app.Use(BotBlocker)

		// Routes
		app.GET("/", HomeHandler)
		app.GET("/about", AboutHandler)

		// public/ is served automatically — robots.txt bypass
		// handled by the path check in BotBlocker above
	}

	return app
}

4. public/robots.txt

Buffalo automatically serves the public/ directory. Because this goes through the same middleware stack, the path check c.Request().URL.Path == "/robots.txt" in BotBlocker ensures the file is always reachable regardless of User-Agent.

# public/robots.txt
# Buffalo serves public/ automatically through its asset pipeline.
# BotBlocker bypasses detection for /robots.txt path.

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

5. Scoped middleware with route groups

Buffalo supports route groups via app.Group(). Apply BotBlocker to a group instead of the whole app when some routes (health checks, webhooks) should remain open to all callers.

// Scoped middleware — apply BotBlocker to specific route groups only
func App() *buffalo.App {
	if app == nil {
		app = buffalo.New(buffalo.Options{Env: ENV})

		// Public routes — no bot blocking
		app.GET("/robots.txt", RobotsHandler)
		app.GET("/health",     HealthHandler)

		// Protected routes — bot blocking applied to this group only
		protected := app.Group("/")
		protected.Use(BotBlocker)
		protected.GET("/",        HomeHandler)
		protected.GET("/blog",    BlogHandler)
		protected.GET("/api/v1/", APIHandler)
	}
	return app
}

Key points

• Set header first: Call c.Response().Header().Set() before any branch. Once a handler writes a body, headers are locked — setting them after next(c) is too late.
• Short-circuit: return c.Render(http.StatusForbidden, r.String("Forbidden")) — do not call next(c). Buffalo's renderer writes the response and returns to the caller.
• Header access: c.Request().Header.Get("User-Agent") — returns the first value for the header, or empty string if absent. Header name lookup is case-insensitive per HTTP/1.1 spec.
• Path access: c.Request().URL.Path — the unescaped path component. Use this (not c.Request().RequestURI) to compare against literal strings like "/robots.txt".
• Route groups: app.Group() creates an isolated middleware stack. Prefer group-scoped middleware over global app.Use() when protecting only a subset of routes.
• r variable: r is Buffalo's render engine, typically initialised in render.go as var r = render.New(render.Options{...}). Use r.String() for plain-text responses.

Framework comparison — Go ecosystem

Buffalo's middleware signature is conceptually the same as Go stdlib (func(http.Handler) http.Handler) but wraps the context object as buffalo.Context for richer request inspection and Buffalo-specific rendering. The key difference from Gin/Echo: Buffalo uses c.Render() (which invokes the render engine) rather than a direct write.

Dependencies

No additional packages needed beyond Buffalo itself. All pattern matching uses Go stdlib strings. Install Buffalo with the CLI:

# Install Buffalo CLI
go install github.com/gobuffalo/cli/cmd/buffalo@latest

# Generate a new app
buffalo new myapp --db-type sqlite3

# Run in development
buffalo dev