Skip to content
TypeScript · Qwik City · Resumable Framework

How to Block AI Bots on Qwik (Qwik City)

Qwik is a resumable JavaScript framework by Builder.io that eliminates hydration by serialising component state into HTML. Qwik City is its meta-framework — file-based routing, server-side middleware via onRequest handlers, and deployment adapters for Node, Cloudflare Pages, Netlify, Vercel, and static SSG. Because Qwik City runs server middleware natively, AI bot protection combines robots.txt, noai meta tags via DocumentHead, response headers, and onRequest middleware for hard 403 blocking.

8 min readUpdated April 2026Qwik 2.x / Qwik City

1. robots.txt

Qwik City serves everything in the public/ directory as static assets. The simplest approach is a static file. For dynamic content that changes based on environment, use a route endpoint.

Static robots.txt

Create public/robots.txt:

# Block all AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow legitimate search crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /

The file is served at yoursite.com/robots.txt automatically — no configuration needed.

Dynamic route endpoint

For environment-specific robots.txt, create a route endpoint at src/routes/robots.txt/index.ts:

// src/routes/robots.txt/index.ts
import type { RequestHandler } from "@builder.io/qwik-city";

export const onGet: RequestHandler = async ({ send, env }) => {
  const isProduction = env.get("NODE_ENV") === "production";

  const rules = isProduction
    ? `User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /`
    : `# Staging — block all crawlers
User-agent: *
Disallow: /`;

  send(
    new Response(rules, {
      status: 200,
      headers: { "content-type": "text/plain" },
    })
  );
};
Route endpoint vs static file: If both public/robots.txt and src/routes/robots.txt/index.ts exist, the route endpoint takes precedence in server mode. In static/SSG mode, the static file wins because there is no server to handle the route. Remove the static file if you use the endpoint approach.
onGet vs onRequest: Use onGet for endpoints that only respond to GET requests (like robots.txt). Use onRequest for middleware that should run on every HTTP method. Both receive the same RequestEvent object.

2. noai meta tags via DocumentHead

Qwik City manages document <head> through the DocumentHead type — either a static object or a function that receives route data. Export head from layout files for site-wide defaults or from individual routes for per-page overrides.

Site-wide default in root layout

Edit src/routes/layout.tsx to add a default noai meta tag:

// src/routes/layout.tsx
import { component$, Slot } from "@builder.io/qwik";
import type { DocumentHead } from "@builder.io/qwik-city";

export default component$(() => {
  return (
    <>
      <header>
        {/* Your header/nav */}
      </header>
      <main>
        <Slot />
      </main>
      <footer>
        {/* Your footer */}
      </footer>
    </>
  );
});

export const head: DocumentHead = {
  meta: [
    {
      name: "robots",
      content: "noai, noimageai",
    },
  ],
};

This applies noai, noimageai to every page that uses this layout. Nested layouts and individual routes can override it.

Per-route override

To allow indexing on a specific page, export head from that route:

// src/routes/blog/index.tsx
import { component$ } from "@builder.io/qwik";
import type { DocumentHead } from "@builder.io/qwik-city";

export default component$(() => {
  return <div>{/* Blog listing */}</div>;
});

export const head: DocumentHead = {
  title: "Blog",
  meta: [
    {
      name: "robots",
      content: "index, follow",
    },
  ],
};
Head resolution order: Qwik City merges head exports from layouts and routes. For meta tags with the same name attribute, the most specific (deepest route) wins. So a route-level robots meta tag overrides the layout-level default — no manual conflict handling needed.

Dynamic head with route data

Use a DocumentHead function to set meta tags based on loaded data:

// src/routes/blog/[slug]/index.tsx
import type { DocumentHead } from "@builder.io/qwik-city";

export const head: DocumentHead = ({ resolveValue }) => {
  const post = resolveValue(usePostData);
  return {
    title: post?.title ?? "Blog Post",
    meta: [
      {
        name: "robots",
        content: post?.allowAITraining
          ? "index, follow"
          : "noai, noimageai",
      },
    ],
  };
};

This lets you control AI bot access at the content level — useful for sites where some content is open and other content is protected.

3. X-Robots-Tag via onRequest middleware

Qwik City's onRequest middleware runs on the server before page rendering. Use it to set HTTP response headers on every request.

Global middleware via plugin file

Create src/routes/plugin@headers.ts:

// src/routes/plugin@headers.ts
import type { RequestHandler } from "@builder.io/qwik-city";

export const onRequest: RequestHandler = async ({ headers, next }) => {
  headers.set("X-Robots-Tag", "noai, noimageai");
  headers.set("X-Content-Type-Options", "nosniff");
  headers.set("X-Frame-Options", "SAMEORIGIN");
  await next();
};
Plugin files: Files named plugin@name.ts in Qwik City act as middleware for all routes in that directory and its subdirectories. A plugin at src/routes/plugin@headers.ts applies to every route in the app. Multiple plugins run in alphabetical order by filename.

Directory-scoped headers

To apply headers only to a specific section, place the plugin in that directory:

// src/routes/api/plugin@api-headers.ts
import type { RequestHandler } from "@builder.io/qwik-city";

export const onRequest: RequestHandler = async ({ headers, next }) => {
  // API routes: block all bot indexing
  headers.set("X-Robots-Tag", "noindex, nofollow, noai, noimageai");
  await next();
};

This only affects routes under /api/. Other routes still use the root-level plugin headers.

Per-route header override

Export onRequest directly from a route file to override or extend headers for that route:

// src/routes/public-content/index.tsx
import type { RequestHandler } from "@builder.io/qwik-city";

export const onRequest: RequestHandler = async ({ headers, next }) => {
  // This specific page allows AI indexing
  headers.set("X-Robots-Tag", "index, follow");
  await next();
};
Header precedence: If both a plugin and a route export onRequest, the plugin runs first (it calls next() to continue). The route's onRequest runs next. The last headers.set() for the same header key wins. So a route-level X-Robots-Tag overrides the plugin-level value.

4. Hard 403 via plugin middleware

A hard 403 blocks the AI bot before any HTML is rendered or serialised — the most effective protection because the bot never sees the content.

Global bot blocker plugin

Create src/routes/plugin@bot-block.ts:

// src/routes/plugin@bot-block.ts
import type { RequestHandler } from "@builder.io/qwik-city";

const AI_BOTS = [
  "GPTBot", "ClaudeBot", "Claude-Web", "anthropic-ai",
  "CCBot", "Google-Extended", "PerplexityBot",
  "Applebot-Extended", "Amazonbot", "meta-externalagent",
  "Bytespider", "DuckAssistBot", "YouBot",
  "Diffbot", "cohere-ai", "OAI-SearchBot",
];

export const onRequest: RequestHandler = async ({
  request,
  send,
  next,
}) => {
  const ua = request.headers.get("user-agent") ?? "";
  const isAIBot = AI_BOTS.some((bot) => ua.includes(bot));

  if (isAIBot) {
    send(
      new Response("Forbidden", {
        status: 403,
        headers: { "content-type": "text/plain" },
      })
    );
    return;
  }

  await next();
};
send() vs throw: Calling send() in Qwik City short-circuits the middleware chain and sends the response immediately — no further middleware or page rendering runs. Do not call next() after send(). This is different from Express-style middleware where you'd use res.status(403).end().

Selective protection

To only protect certain routes, use path-based conditions:

// src/routes/plugin@bot-block.ts
import type { RequestHandler } from "@builder.io/qwik-city";

const AI_BOTS = [
  "GPTBot", "ClaudeBot", "CCBot", "Google-Extended",
  "PerplexityBot", "Bytespider",
];

const PROTECTED_PATHS = ["/premium", "/members", "/api"];

export const onRequest: RequestHandler = async ({
  request,
  url,
  send,
  next,
}) => {
  const ua = request.headers.get("user-agent") ?? "";
  const isAIBot = AI_BOTS.some((bot) => ua.includes(bot));
  const isProtected = PROTECTED_PATHS.some((p) =>
    url.pathname.startsWith(p)
  );

  if (isAIBot && isProtected) {
    send(
      new Response("Forbidden", {
        status: 403,
        headers: { "content-type": "text/plain" },
      })
    );
    return;
  }

  await next();
};

Combined: headers + 403

You can combine X-Robots-Tag headers and hard 403 in a single plugin. Headers protect well-behaved bots; the 403 stops those that ignore robots signals:

// src/routes/plugin@ai-protection.ts
import type { RequestHandler } from "@builder.io/qwik-city";

const BLOCK_403 = [
  "GPTBot", "ClaudeBot", "CCBot", "Bytespider",
  "anthropic-ai", "meta-externalagent",
];

export const onRequest: RequestHandler = async ({
  request,
  headers,
  send,
  next,
}) => {
  const ua = request.headers.get("user-agent") ?? "";

  // Hard block known aggressive crawlers
  if (BLOCK_403.some((bot) => ua.includes(bot))) {
    send(new Response("Forbidden", {
      status: 403,
      headers: { "content-type": "text/plain" },
    }));
    return;
  }

  // Signal to all other bots via header
  headers.set("X-Robots-Tag", "noai, noimageai");
  await next();
};

5. Static adapter (SSG) considerations

Qwik City's static adapter pre-renders all routes to HTML files at build time. There is no server runtime, so onRequest middleware does not execute in production. You must handle bot blocking at the hosting layer instead.

What works in SSG mode

What requires hosting-layer support

Netlify headers for SSG

# netlify.toml
[build]
  command = "npm run build"
  publish = "dist"

[[headers]]
  for = "/*"
  [headers.values]
    X-Robots-Tag = "noai, noimageai"

Vercel headers for SSG

{
  "headers": [
    {
      "source": "/(.*)",
      "headers": [
        {
          "key": "X-Robots-Tag",
          "value": "noai, noimageai"
        }
      ]
    }
  ]
}

Cloudflare Pages headers for SSG

Create public/_headers (copied to the output directory during build):

/*
  X-Robots-Tag: noai, noimageai
SSG trade-off: If you use the static adapter, you lose server-side middleware entirely. For sites that need fine-grained, route-level AI bot control, use the Node, Cloudflare Pages, or Netlify adapter instead — these run your onRequest handlers in production.

6. Deployment comparison

Qwik City's adapter system determines the runtime. Here is how each adapter and host handles AI bot protection:

Adapter / Hostrobots.txtnoai metaX-Robots-TagHard 403
Node (Express/Fastify)public/ static ✓DocumentHead ✓onRequest plugin ✓onRequest plugin ✓
Cloudflare Pagespublic/ static ✓DocumentHead ✓onRequest plugin ✓onRequest plugin ✓
Netlify Edgepublic/ static ✓DocumentHead ✓onRequest plugin ✓onRequest plugin ✓
Vercel Edgepublic/ static ✓DocumentHead ✓onRequest plugin ✓onRequest plugin ✓
Denopublic/ static ✓DocumentHead ✓onRequest plugin ✓onRequest plugin ✓
Static (SSG)public/ static ✓DocumentHead (build-time) ✓Host config only ✗*Host Edge Functions only ✗*

✗* = onRequest plugins do not run in SSG mode. Use hosting-layer configuration (see Section 5).

Every server-side adapter (Node, Cloudflare, Netlify, Vercel, Deno) supports all four layers of AI bot protection natively through Qwik City's middleware system. The static adapter requires hosting-layer configuration for headers and hard 403.

Qwik City project structure with AI protection

your-qwik-app/
├── public/
│   └── robots.txt                    # Static robots.txt
├── src/
│   └── routes/
│       ├── layout.tsx                # Default noai meta via DocumentHead
│       ├── plugin@bot-block.ts       # Hard 403 for AI bots
│       ├── plugin@headers.ts         # X-Robots-Tag header
│       ├── robots.txt/
│       │   └── index.ts              # Dynamic robots.txt endpoint (optional)
│       ├── index.tsx                  # Homepage
│       └── blog/
│           ├── layout.tsx            # Blog-specific head (override noai)
│           └── [slug]/
│               └── index.tsx         # Per-post head via resolveValue
├── adapters/
│   └── cloudflare-pages/             # Adapter config
│       └── vite.config.ts
└── package.json

FAQ

How do I add robots.txt to a Qwik City site?

Place robots.txt in the public/ directory — Qwik City serves everything in public/ as static assets. For dynamic content, create a route endpoint at src/routes/robots.txt/index.ts that returns a text/plain Response.

How do I add noai meta tags in Qwik City?

Export a head object from your layout or route file:

export const head: DocumentHead = {
  meta: [
    { name: "robots", content: "noai, noimageai" },
  ],
};

Layout-level defaults apply to all child routes. Per-route exports override the layout's meta tags when they share the same name attribute.

What is onRequest in Qwik City?

onRequest is a server-side middleware handler exported from route files or plugin files. It runs before the page renders and receives a RequestEvent with access to headers, URL, query params, and a next() function. Plugin files (plugin@name.ts) apply middleware to all routes under that directory.

How do plugin files work in Qwik City?

Files named plugin@name.ts act as middleware for their directory and all subdirectories. Place them at src/routes/ for global scope, or in a subdirectory for scoped middleware. Multiple plugins run in alphabetical order by filename. Each must call await next() to continue the chain (unless short-circuiting with send()).

Does blocking AI bots affect Qwik's resumability?

No. AI bot blocking happens at the server middleware layer before any HTML is sent. Qwik's resumability, lazy loading, and qwikloader.js are client-side concerns that execute in browsers. Blocked bots never receive the HTML, so there is no impact on client-side behaviour.

Will blocking AI bots affect my SEO?

Blocking AI-specific crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) does not affect standard search engine indexing. Googlebot and Bingbot are separate user agents from Google-Extended and are not blocked by the configurations in this guide. Always include explicit Allow rules for Googlebot and Bingbot in your robots.txt.

Is your site protected from AI bots?

Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.

Related Guides

How to Block AI Bots on Next.js: Complete 2026 GuideHow to Block AI Bots on SvelteKit: Complete 2026 GuideHow to Block AI Bots on SolidStart: Complete 2026 Guiderobots.txt for AI Bots: Complete 2026 GuideHow to Block GPTBot (OpenAI)