How to Block AI Bots on Kong Gateway: Complete 2026 Guide

Step 1 — Plugin handler (handler.lua)

Two phases: access() for blocking (before upstream) and header_filter() for adding X-Robots-Tag to all legitimate responses. Lua patterns use %- for literal hyphens — different from regex.

-- kong/plugins/ai-bot-blocker/handler.lua

local AiBotBlocker = {}

AiBotBlocker.PRIORITY = 1000   -- Run early; higher = earlier
AiBotBlocker.VERSION  = "1.0.0"

-- AI bot User-Agent patterns (lowercase for case-insensitive matching)
local AI_BOT_PATTERNS = {
  -- OpenAI
  "gptbot", "chatgpt%-user", "oai%-searchbot",
  -- Anthropic
  "claudebot", "claude%-web",
  -- Common Crawl
  "ccbot",
  -- Bytedance
  "bytespider",
  -- Meta
  "meta%-externalagent",
  -- Perplexity
  "perplexitybot",
  -- Google AI
  "google%-extended", "googleother",
  -- Cohere
  "cohere%-ai",
  -- Amazon
  "amazonbot",
  -- Diffbot
  "diffbot",
  -- AI2
  "ai2bot",
  -- DeepSeek
  "deepseekbot",
  -- Mistral
  "mistralai%-user",
  -- xAI
  "xai%-bot",
  -- You.com
  "youbot",
  -- DuckDuckGo AI
  "duckassistbot",
}

-- Note: Lua patterns use % to escape special chars (%-  = literal hyphen)

local function is_ai_bot(user_agent)
  if not user_agent or user_agent == "" then
    return false
  end
  local ua = user_agent:lower()
  for _, pattern in ipairs(AI_BOT_PATTERNS) do
    if ua:find(pattern) then
      return true
    end
  end
  return false
end

-- access() phase: runs BEFORE forwarding to upstream.
-- Call kong.response.exit() to block — upstream never receives the request.
function AiBotBlocker:access(config)
  -- Allow robots.txt regardless of User-Agent
  if kong.request.get_path() == "/robots.txt" then
    return
  end

  local ua = kong.request.get_header("User-Agent") or ""

  if is_ai_bot(ua) then
    -- Exit with 403 — upstream never called
    return kong.response.exit(403, "Forbidden", {
      ["Content-Type"]  = "text/plain; charset=utf-8",
      ["X-Robots-Tag"]  = "noai, noimageai",
    })
  end
end

-- header_filter() phase: runs on responses from upstream.
-- Add X-Robots-Tag to all non-blocked (legitimate) responses.
function AiBotBlocker:header_filter(config)
  kong.response.add_header("X-Robots-Tag", "noai, noimageai")
end

return AiBotBlocker

Step 2 — Plugin schema (schema.lua)

The schema defines config fields exposed via Admin API and declarative YAML. allow_paths defaults to ["/robots.txt", "/health"] — these paths bypass the bot check even with the plugin enabled globally.

-- kong/plugins/ai-bot-blocker/schema.lua
-- Schema defines the plugin's config fields (optional custom patterns).

local typedefs = require "kong.db.schema.typedefs"

local PLUGIN_NAME = "ai-bot-blocker"

local schema = {
  name = PLUGIN_NAME,
  fields = {
    -- Standard Kong field — which protocols this plugin runs on
    { protocols = typedefs.protocols_http },
    { config = {
        type = "record",
        fields = {
          -- Optional: additional custom patterns to block
          -- If empty, only built-in patterns are used.
          {
            extra_patterns = {
              type     = "array",
              elements = { type = "string" },
              default  = {},
            }
          },
          -- Optional: paths to always allow (bypass the blocker)
          {
            allow_paths = {
              type     = "array",
              elements = { type = "string" },
              default  = { "/robots.txt", "/health" },
            }
          },
        },
      },
    },
  },
}

return schema

Step 3 — Plugin installation

Kong loads plugins from the Lua package path. Set KONG_PLUGINS=bundled,ai-bot-blocker (or in kong.conf) to enable. In Kubernetes, mount the plugin files via ConfigMap.

# Plugin installation — three methods

# Method 1: Copy directly to Kong's plugin directory
# Kong reads plugins from KONG_PLUGINS and KONG_LUA_PACKAGE_PATH
mkdir -p /usr/local/share/lua/5.1/kong/plugins/ai-bot-blocker
cp handler.lua /usr/local/share/lua/5.1/kong/plugins/ai-bot-blocker/
cp schema.lua  /usr/local/share/lua/5.1/kong/plugins/ai-bot-blocker/

# Set environment variable to enable the plugin
export KONG_PLUGINS="bundled,ai-bot-blocker"
# Or in kong.conf:
# plugins = bundled,ai-bot-blocker

# Method 2: Docker — mount plugin directory
# docker run -d \
#   -v $(pwd)/plugins:/usr/local/share/lua/5.1/kong/plugins \
#   -e KONG_PLUGINS="bundled,ai-bot-blocker" \
#   -e KONG_DATABASE=off \
#   -e KONG_DECLARATIVE_CONFIG=/kong/declarative/kong.yml \
#   kong:3.6

# Method 3: Helm (Kong for Kubernetes)
# values.yaml:
# plugins:
#   configMaps:
#   - name: kong-plugin-ai-bot-blocker
#     pluginName: ai-bot-blocker

Step 4 — Enable via Admin API (global, service, or route scope)

Admin API changes take effect immediately — no Kong restart needed. More specific scopes (route > service > global) take precedence when multiple plugins of the same name exist.

# Enable via Kong Admin API

# Global scope — applies to ALL services and routes
curl -X POST http://localhost:8001/plugins \
  --data "name=ai-bot-blocker"

# Per-service scope — applies to all routes of a specific service
curl -X POST http://localhost:8001/services/{service-id}/plugins \
  --data "name=ai-bot-blocker"

# Per-route scope — applies to a specific route only
curl -X POST http://localhost:8001/routes/{route-id}/plugins \
  --data "name=ai-bot-blocker"

# With custom config — extra patterns and allowed paths
curl -X POST http://localhost:8001/plugins \
  --data "name=ai-bot-blocker" \
  --data "config.extra_patterns[]=mybot" \
  --data "config.allow_paths[]=/robots.txt" \
  --data "config.allow_paths[]=/health" \
  --data "config.allow_paths[]=/sitemap.xml"

# Verify the plugin was created
curl http://localhost:8001/plugins | jq '.data[] | select(.name == "ai-bot-blocker")'

Step 5 — Declarative YAML (kong.yml / deck)

For GitOps workflows, define the plugin in declarative YAML and apply with deck sync. DB-less mode uses KONG_DECLARATIVE_CONFIG pointing to the file directly.

# kong.yml — declarative configuration (DB-less mode or deck sync)
# Use 'deck sync' to apply to a running Kong instance.

_format_version: "3.0"

services:
  - name: my-api
    url: http://my-upstream:3000

    routes:
      # Regular API routes — bot-blocker applies via global plugin
      - name: api-routes
        paths:
          - /api
        strip_path: false

      # robots.txt route — NO plugins, always accessible
      - name: robots-txt
        paths:
          - /robots.txt
        strip_path: false
        # No plugins key here — global ai-bot-blocker does NOT apply
        # because route-level plugin absence + allow_paths in config handles it

# Global plugin — applies to all services and routes
plugins:
  - name: ai-bot-blocker
    config:
      allow_paths:
        - /robots.txt
        - /health
        - /sitemap.xml

  # Optional: Response transformer for belt-and-suspenders X-Robots-Tag
  # (handler.lua header_filter handles this, but useful as a fallback)
  # - name: response-transformer
  #   config:
  #     add:
  #       headers:
  #         - "X-Robots-Tag: noai, noimageai"

Step 6 — robots.txt via Kong

The request-termination plugin (built-in) can serve robots.txt content directly from Kong — no upstream needed for this path. Importantly, request-termination fires before other plugins, so AI bots can always read robots.txt.

# robots.txt via Kong — two approaches

# Approach 1: Serve robots.txt from an upstream service
# Just ensure /robots.txt is in the allow_paths config so the
# ai-bot-blocker doesn't intercept it before forwarding to upstream.
# The upstream service (nginx, Express, etc.) serves the file.

# Approach 2: Kong Request Termination plugin — serve robots.txt
# directly from Kong without a backend (no upstream needed)
curl -X POST http://localhost:8001/routes/{robots-route-id}/plugins \
  --data "name=request-termination" \
  --data "config.status_code=200" \
  --data "config.content_type=text/plain; charset=utf-8" \
  --data 'config.body=User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: AmazonBot
Disallow: /'

# Note: the request-termination plugin DOES NOT trigger other plugins
# including ai-bot-blocker — so AI bots can always read robots.txt.
# Register the robots.txt route BEFORE enabling ai-bot-blocker globally,
# or use allow_paths in the plugin config.

Kong custom plugin vs IP Restriction vs Nginx vs HAProxy