How to Block AI Bots on SolidStart: Complete 2026 Guide

Methods at a glance

1. robots.txt — public/ directory

Drop robots.txt in public/. Vinxi copies everything in public/ verbatim to the build output — no configuration needed. The file is served at /robots.txt before any middleware or route handlers run.

public/robots.txt

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Dynamic robots.txt via API route (optional)

For environment-based rules (stricter in staging), use a src/routes/robots.txt.ts API route instead of the static file. Delete public/robots.txt first — static assets take precedence over route handlers in SolidStart.

// src/routes/robots.txt.ts
import { APIEvent } from "@solidjs/start/server";

export function GET(_event: APIEvent) {
  const isProd = process.env.NODE_ENV === "production";

  const rules = isProd
    ? `User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: *
Allow: /`
    : "User-agent: *
Disallow: /"; // Block all bots in staging

  return new Response(rules, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

2. Hard 403 blocking via middleware

createMiddleware runs on every request before routing — the correct place for bot blocking. Module-level regex is compiled once at startup, not per-request. Always exempt /robots.txt first.

src/middleware.ts

import { createMiddleware } from "@solidjs/start/middleware";

// Compiled once at module load — not per-request
const BLOCKED_UAS = /GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-Web|anthropic-ai|Google-Extended|Bytespider|CCBot|PerplexityBot|Applebot-Extended|meta-externalagent|Diffbot|ImagesiftBot|FacebookBot|YouBot|PerplexityBot/i;

export default createMiddleware({
  onRequest: [
    (event) => {
      const url = new URL(event.request.url);

      // Always serve robots.txt — never block it
      if (url.pathname === "/robots.txt") return;

      const ua = event.request.headers.get("user-agent") ?? "";

      if (BLOCKED_UAS.test(ua)) {
        return new Response("Forbidden", {
          status: 403,
          headers: {
            "Content-Type": "text/plain",
            "X-Robots-Tag": "noai, noimageai",
          },
        });
      }
    },
  ],
});

app.config.ts — register the middleware

// app.config.ts
import { defineConfig } from "@solidjs/start/config";

export default defineConfig({
  middleware: "./src/middleware.ts",
  // other options...
});

Adding X-Robots-Tag to all responses

// src/middleware.ts — full version with X-Robots-Tag
import { createMiddleware } from "@solidjs/start/middleware";

const BLOCKED_UAS = /GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-Web|anthropic-ai|Google-Extended|Bytespider|CCBot|PerplexityBot|Applebot-Extended/i;

export default createMiddleware({
  onRequest: [
    // Bot blocking
    (event) => {
      const url = new URL(event.request.url);
      if (url.pathname === "/robots.txt") return;

      const ua = event.request.headers.get("user-agent") ?? "";
      if (BLOCKED_UAS.test(ua)) {
        return new Response("Forbidden", {
          status: 403,
          headers: { "Content-Type": "text/plain" },
        });
      }
    },
  ],
  onBeforeResponse: [
    // Add X-Robots-Tag to all HTML responses
    (_event, response) => {
      const ct = response.headers.get("content-type") ?? "";
      if (ct.includes("text/html")) {
        response.headers.set("X-Robots-Tag", "noai, noimageai");
      }
    },
  ],
});

3. noai meta tags — @solidjs/meta

SolidStart uses @solidjs/meta for head management. With SSR enabled (the default), <Meta> tags are included in the server-rendered HTML — non-JS crawlers see them on the initial response.

src/app.tsx — global noai meta tag

// src/app.tsx
import { MetaProvider, Meta, Title } from "@solidjs/meta";
import { Router } from "@solidjs/router";
import { FileRoutes } from "@solidjs/start/router";
import { Suspense } from "solid-js";
import "./app.css";

export default function App() {
  return (
    <Router
      root={(props) => (
        <MetaProvider>
          {/* Global noai tag — applies to every page */}
          <Meta name="robots" content="noai, noimageai" />
          <Suspense>{props.children}</Suspense>
        </MetaProvider>
      )}
    >
      <FileRoutes />
    </Router>
  );
}

Per-page override

To allow AI training on specific pages (e.g., your public blog), override the global meta tag in the route component. @solidjs/meta deduplicates by name — the last declaration wins.

// src/routes/blog/[slug].tsx — allow training on public posts
import { Meta, Title } from "@solidjs/meta";
import { useParams } from "@solidjs/router";
import { createResource } from "solid-js";

export default function BlogPost() {
  const params = useParams();
  const [post] = createResource(() => fetchPost(params.slug));

  return (
    <>
      {/* Override global noai — allow training on this public post */}
      <Meta name="robots" content="index, follow" />
      <Title>{post()?.title ?? "Blog"}</Title>
      {/* ... */}
    </>
  );
}

// To block AI training on specific pages, omit the override — global noai applies.
// To block AI training on all EXCEPT specific pages, set global to "noai, noimageai"
// and override to "index, follow" only on the pages you want AI to train on.

4. SSR vs SSG vs SPA — what bots see

The rendering mode determines what AI crawlers see. Middleware runs in SSR and at the edge/CDN layer for SSG. SPA mode has no server — bots see an empty HTML shell.

SSG + prerendering: edge middleware for hard blocking

If you prerender pages to static HTML, your src/middleware.ts does not run per-request. Add hard blocking at the hosting platform instead:

// app.config.ts — SSG with prerendering
import { defineConfig } from "@solidjs/start/config";

export default defineConfig({
  // middleware.ts still applies during SSR (dev) but not in static output
  middleware: "./src/middleware.ts",
  server: {
    prerender: {
      crawlLinks: true,
      routes: ["/", "/about", "/blog"],
    },
  },
});

# netlify.toml — hard block at CDN edge for prerendered SolidStart
[[edge_functions]]
  path = "/*"
  function = "block-ai-bots"

# netlify/edge-functions/block-ai-bots.ts
export default async (request: Request) => {
  const url = new URL(request.url);
  if (url.pathname === "/robots.txt") return;

  const ua = request.headers.get("user-agent") ?? "";
  const blocked = /GPTBot|ChatGPT-User|ClaudeBot|Google-Extended|Bytespider|CCBot|PerplexityBot/i;

  if (blocked.test(ua)) {
    return new Response("Forbidden", { status: 403 });
  }
};

5. X-Robots-Tag via hosting platform

For SSR deployments, the middleware approach above handles X-Robots-Tag. For static/SSG deployments, set it via your hosting platform config.

Vercel — vercel.json

{
  "headers": [
    {
      "source": "/(.*)",
      "headers": [
        {
          "key": "X-Robots-Tag",
          "value": "noai, noimageai"
        }
      ]
    }
  ]
}

Netlify — netlify.toml

[[headers]]
  for = "/*"
  [headers.values]
    X-Robots-Tag = "noai, noimageai"

Cloudflare Pages — _headers file

# public/_headers (or dist/_headers for prerendered builds)
/*
  X-Robots-Tag: noai, noimageai

6. Deployment

SolidStart uses Vinxi presets for different deployment targets. The server.preset in app.config.ts determines the output format.

app.config.ts — Vercel preset

import { defineConfig } from "@solidjs/start/config";

export default defineConfig({
  middleware: "./src/middleware.ts",
  server: {
    preset: "vercel",  // or "netlify", "cloudflare-pages", "node", "bun", "deno"
  },
});

// Build: npx vinxi build
// Vercel deploys automatically on git push (auto-detects SolidStart)
// Netlify: add build command "npx vinxi build" and publish dir ".output/public"

Node.js — self-hosted

// app.config.ts
import { defineConfig } from "@solidjs/start/config";

export default defineConfig({
  middleware: "./src/middleware.ts",
  server: {
    preset: "node",
  },
});

// Build and run:
// npx vinxi build
// node .output/server/index.mjs

Docker — multi-stage Node build

# Dockerfile
FROM node:22-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npx vinxi build

FROM node:22-alpine AS runner
WORKDIR /app
COPY --from=builder /app/.output ./
EXPOSE 3000
CMD ["node", "server/index.mjs"]

nginx reverse proxy

If you put nginx in front of SolidStart, you can add a second layer of bot blocking at the proxy level — runs before SolidStart even sees the request.

# /etc/nginx/sites-available/solidstart
map $http_user_agent $blocked_bot {
  default         0;
  "~*GPTBot"      1;
  "~*ChatGPT-User" 1;
  "~*ClaudeBot"   1;
  "~*Claude-Web"  1;
  "~*anthropic-ai" 1;
  "~*Google-Extended" 1;
  "~*Bytespider"  1;
  "~*CCBot"       1;
  "~*PerplexityBot" 1;
}

server {
  listen 80;
  server_name example.com;

  # Never block robots.txt
  location = /robots.txt {
    proxy_pass http://localhost:3000;
  }

  location / {
    if ($blocked_bot) {
      return 403 "Forbidden";
    }

    proxy_pass http://localhost:3000;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}

Frequently asked questions