How to Block AI Bots on Deno: Complete 2026 Guide

Methods at a glance

1. robots.txt — explicit route

Deno does not auto-serve static files. Add a dedicated handler for /robots.txt in your Deno.serve() call. For local development you can read from disk; for Deno Deploy, embed as a constant (covered in section 2).

static/robots.txt

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

server.ts — Deno.serve() with /robots.txt route

// Read at module load time — fails fast if file is missing
// Requires: deno run --allow-net --allow-read=./static server.ts
const ROBOTS_TXT = Deno.readTextFileSync(
  new URL('./static/robots.txt', import.meta.url)
);

Deno.serve({ port: 8000 }, (req: Request): Response => {
  const url = new URL(req.url);

  // Serve robots.txt before any other logic
  if (url.pathname === '/robots.txt') {
    return new Response(ROBOTS_TXT, {
      headers: { 'Content-Type': 'text/plain; charset=utf-8' },
    });
  }

  // ... rest of your handlers
  return new Response('Hello Deno', { status: 200 });
});

import.meta.url resolves relative to the module file, not the current working directory — consistent regardless of where you run deno from.

2. Deno Deploy — embed robots.txt as a constant

Deno Deploy runs code in V8 isolates with no file system access. Reading Deno.readTextFileSync will throw at runtime. Embed the robots.txt content as a TypeScript string constant instead.

server.ts — Deploy-safe embedding

// Embedded — works on Deno Deploy, Deno KV, Deno Cron
const ROBOTS_TXT = `User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: *
Allow: /`;

Deno.serve((req: Request): Response => {
  const { pathname } = new URL(req.url);

  if (pathname === '/robots.txt') {
    return new Response(ROBOTS_TXT, {
      headers: { 'Content-Type': 'text/plain; charset=utf-8' },
    });
  }

  return new Response('Hello', { status: 200 });
});

3. Oak middleware — hard 403 before routes

Oak is the most widely used Deno web framework. Register app.use() before app.use(router.routes()). Oak processes middleware in registration order — a bot-blocking middleware registered first intercepts every request before any route handler runs.

deno.json — Oak dependency

{
  "imports": {
    "@oak/oak": "jsr:@oak/oak@^13"
  }
}

server.ts — Oak with bot-blocking middleware

import { Application, Router } from '@oak/oak';

const BLOCKED_UAS = /GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-Web|anthropic-ai|Google-Extended|Bytespider|CCBot|PerplexityBot|Applebot-Extended/i;

const ROBOTS_TXT = `User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: *
Allow: /`;

const app = new Application();
const router = new Router();

// 1. Serve robots.txt first — exempt from bot blocking
app.use(async (ctx, next) => {
  if (ctx.request.url.pathname === '/robots.txt') {
    ctx.response.type = 'text/plain';
    ctx.response.body = ROBOTS_TXT;
    return; // Do NOT call next() — short-circuit here
  }
  await next();
});

// 2. Block AI bots — registered before router
app.use(async (ctx, next) => {
  const ua = ctx.request.headers.get('user-agent') ?? '';
  if (BLOCKED_UAS.test(ua)) {
    ctx.response.status = 403;
    ctx.response.type = 'text/plain';
    ctx.response.body = 'Forbidden';
    return;
  }
  await next();
});

// 3. Routes — only reached by non-blocked requests
router.get('/', (ctx) => {
  ctx.response.body = 'Hello Oak';
});

app.use(router.routes());
app.use(router.allowedMethods());

await app.listen({ port: 8000 });

4. Raw Deno.serve() — no framework

If you are not using Oak or Fresh, handle everything in the Deno.serve() callback. The pattern is identical to a switch on pathname with an early return for /robots.txt.

// Compiled once at module load — not per request
const BLOCKED_UAS = /GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-Web|anthropic-ai|Google-Extended|Bytespider|CCBot|PerplexityBot|Applebot-Extended/i;

const ROBOTS_TXT = `User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /`;

Deno.serve({ port: 8000 }, (req: Request): Response => {
  const { pathname } = new URL(req.url);

  // Always serve robots.txt — exempt from blocking
  if (pathname === '/robots.txt') {
    return new Response(ROBOTS_TXT, {
      headers: { 'Content-Type': 'text/plain; charset=utf-8' },
    });
  }

  // Block AI crawlers on all other paths
  const ua = req.headers.get('user-agent') ?? '';
  if (BLOCKED_UAS.test(ua)) {
    return new Response('Forbidden', { status: 403 });
  }

  // Normal request handling
  if (pathname === '/') {
    return new Response('Hello Deno', {
      headers: { 'Content-Type': 'text/plain' },
    });
  }

  return new Response('Not Found', { status: 404 });
});

Run with: deno run --allow-net server.ts. No --allow-read needed when the robots.txt is embedded.

5. noai meta tag in HTML responses

Deno has no templating engine by default — HTML is a string. Add the noai meta tag to your base HTML string. For per-page control, pass a boolean flag to your template function.

Template with noai support

function renderPage(
  content: string,
  opts: { blockAiTraining?: boolean } = {}
): Response {
  const robotsMeta = opts.blockAiTraining
    ? '<meta name="robots" content="noai, noimageai">'
    : '';

  const html = `<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  ${robotsMeta}
  <title>My Deno App</title>
</head>
<body>
  ${content}
</body>
</html>`;

  return new Response(html, {
    headers: { 'Content-Type': 'text/html; charset=utf-8' },
  });
}

// Usage — block AI training on specific pages
Deno.serve((req) => {
  const { pathname } = new URL(req.url);

  if (pathname === '/') {
    return renderPage('<h1>Home</h1>', { blockAiTraining: true });
  }

  if (pathname === '/public-page') {
    // No blockAiTraining — AI training allowed on this page
    return renderPage('<h1>Public</h1>');
  }

  return new Response('Not Found', { status: 404 });
});

noai signals crawlers not to use this page for AI training. noimageai applies the same restriction to images on the page. These are voluntary signals — use the UA-blocking middleware from section 3 for enforcement.

6. X-Robots-Tag response header

X-Robots-Tag in the HTTP response header applies to all content types including PDFs, images, and JSON APIs — unlike the meta tag which only applies to HTML pages that bots parse.

Oak — global X-Robots-Tag middleware

// Add after the bot-blocking middleware, before routes
app.use(async (ctx, next) => {
  await next();
  // Set on the way out — after the route has run
  ctx.response.headers.set('X-Robots-Tag', 'noai, noimageai');
});

Raw Deno.serve() — wrap Response headers

function withRobotsHeader(res: Response): Response {
  const headers = new Headers(res.headers);
  headers.set('X-Robots-Tag', 'noai, noimageai');
  return new Response(res.body, { status: res.status, headers });
}

Deno.serve((req) => {
  const { pathname } = new URL(req.url);

  // Never set X-Robots-Tag on robots.txt itself
  if (pathname === '/robots.txt') {
    return new Response(ROBOTS_TXT, {
      headers: { 'Content-Type': 'text/plain' },
    });
  }

  const ua = req.headers.get('user-agent') ?? '';
  if (BLOCKED_UAS.test(ua)) {
    return new Response('Forbidden', { status: 403 });
  }

  return withRobotsHeader(
    new Response('<h1>Hello</h1>', {
      headers: { 'Content-Type': 'text/html' },
    })
  );
});

7. Fresh — routes/_middleware.ts

Fresh is Deno's official full-stack meta-framework. Place a _middleware.ts file in the routes/ directory — Fresh runs it for every request before any page route handler executes.

routes/_middleware.ts (Fresh 1.x)

import { FreshContext } from '$fresh/server.ts';

const BLOCKED_UAS = /GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-Web|anthropic-ai|Google-Extended|Bytespider|CCBot|PerplexityBot|Applebot-Extended/i;

export async function handler(
  req: Request,
  ctx: FreshContext
): Promise<Response> {
  // Exempt /robots.txt from blocking
  const { pathname } = new URL(req.url);
  if (pathname === '/robots.txt') {
    return ctx.next();
  }

  const ua = req.headers.get('user-agent') ?? '';
  if (BLOCKED_UAS.test(ua)) {
    return new Response('Forbidden', { status: 403 });
  }

  return ctx.next();
}

routes/_middleware.ts (Fresh 2.x)

import { define } from '../utils.ts'; // or your Fresh 2 define helper

const BLOCKED_UAS = /GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-Web|anthropic-ai|Google-Extended|Bytespider|CCBot|PerplexityBot|Applebot-Extended/i;

export const handler = define.middleware(async (ctx) => {
  const { pathname } = new URL(ctx.req.url);

  if (pathname !== '/robots.txt') {
    const ua = ctx.req.headers.get('user-agent') ?? '';
    if (BLOCKED_UAS.test(ua)) {
      return new Response('Forbidden', { status: 403 });
    }
  }

  return ctx.next();
});

routes/robots.txt.ts — Fresh robots.txt route

// Fresh uses file-based routing — routes/robots.txt.ts → GET /robots.txt
import { Handlers } from '$fresh/server.ts';

const ROBOTS = `User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: *
Allow: /`;

export const handler: Handlers = {
  GET() {
    return new Response(ROBOTS, {
      headers: { 'Content-Type': 'text/plain' },
    });
  },
};

Fresh's file-based router maps routes/robots.txt.ts to GET /robots.txt. The _middleware.ts handler checks for this path and calls ctx.next() without blocking — the route returns the robots file normally.

8. Deno permissions and bot blocking

Deno's permission model is a security layer independent of bot blocking — but the two interact. Scope your permissions as narrowly as possible.

9. nginx — reverse proxy hard block

nginx sits in front of Deno and blocks matched bots before any request reaches your process. This is the most reliable approach — bots get a 403 at the network edge, your Deno process never handles them.

map $http_user_agent $block_ai_bot {
    default         0;
    "~*GPTBot"      1;
    "~*ChatGPT-User" 1;
    "~*OAI-SearchBot" 1;
    "~*ClaudeBot"   1;
    "~*Claude-Web"  1;
    "~*anthropic-ai" 1;
    "~*Google-Extended" 1;
    "~*Bytespider"  1;
    "~*CCBot"       1;
    "~*PerplexityBot" 1;
    "~*Applebot-Extended" 1;
}

server {
    listen 80;
    server_name yourdomain.com;

    # Always serve robots.txt directly — before bot check
    location = /robots.txt {
        alias /var/www/static/robots.txt;
        add_header Content-Type text/plain;
    }

    location / {
        if ($block_ai_bot) {
            return 403;
        }
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

10. Docker deployment

Deno's official Docker image is lean. Use the embedded-constant approach (section 2) to avoid needing --allow-read in production.

FROM denoland/deno:2.2.0

WORKDIR /app

# Cache dependencies first (deno.json + deno.lock)
COPY deno.json deno.lock ./
RUN deno install --frozen

# Copy source — use embedded robots.txt constant, no static/ needed
COPY . .

# Cache the entry point
RUN deno cache server.ts

EXPOSE 8000

# --allow-net only — no file system access needed with embedded constant
CMD ["deno", "run", "--allow-net", "server.ts"]

If you use the file-based robots.txt approach, add COPY static/ ./static/ before COPY . . and add --allow-read=./static to the CMD.

Deno runtime comparison

FAQ