vite-plugin-robots-txt

Generate a robots.txt for any Vite-powered app (React, SvelteKit, Vue, Solid, Preact, vanilla) with zero framework assumptions.

• 🧩 Framework-agnostic — works with any Vite project
• 📝 Simple API — pass policies or a dynamic policyBuilder
• 🗺️ Sitemaps support — static or generated
• 🧪 Dev-friendly — serves /robots.txt with Cache-Control: no-store
• 📦 Build output — writes robots.txt into your static folder (auto-detected)

Neutral by default: if you don’t provide any options, the plugin generates:

User-agent: *
Allow: /

Requirements

• Node: >= 18
• Vite: >= 4 (tested on Vite 5)

Install

# choose one:
npm i -D vite-plugin-robots-txt
# or
pnpm add -D vite-plugin-robots-txt
# or (scoped)
npm i -D vite-plugin-robots-txt

Quick start

// vite.config.ts
import {defineConfig} from "vite";
import react from "@vitejs/plugin-react";
import generateRobotsTxt from "vite-plugin-robots-txt";

export default defineConfig({
    plugins: [
        react(),
        generateRobotsTxt(),
    ],
});

This creates/serves a neutral robots.txt:

User-agent: *
Allow: /

• In dev (vite serve), the plugin serves /robots.txt with Cache-Control: no-store so changes appear immediately.
• In build, the plugin writes robots.txt to your static dir (auto-detected: prefers static/ for SvelteKit, otherwise public/).

Framework examples

SvelteKit

import {defineConfig} from "vite";
import {sveltekit} from "@sveltejs/kit/vite";
import generateRobotsTxt from "vite-plugin-robots-txt";

export default defineConfig({
    plugins: [
        sveltekit(),
        generateRobotsTxt({
            policies: [{userAgent: "*", allow: ["/"], disallow: ["/admin"]}],
            sitemaps: ({mode}) =>
                mode === "production" ? ["https://example.com/sitemap.xml"] : [],
            footerComment: ({mode}) => `${mode} robots generated by vite-plugin-robots-txt`,
        }),
    ],
});

React (Vite)

import {defineConfig} from "vite";
import react from "@vitejs/plugin-react";
import generateRobotsTxt from "vite-plugin-robots-txt";

export default defineConfig({
    plugins: [
        react(),
        generateRobotsTxt({
            // Will default to "public" for output
            policies: [
                {userAgent: "*", allow: ["/"], disallow: ["/private", "/preview"]},
            ],
            sitemaps: ["https://example.com/sitemap.xml"],
        }),
    ],
});

Vue (Vite)

import {defineConfig} from "vite";
import vue from "@vitejs/plugin-vue";
import generateRobotsTxt from "vite-plugin-robots-txt";

export default defineConfig({
    plugins: [
        vue(),
        generateRobotsTxt({
            policies: [
                {userAgent: "*", allow: ["/"], disallow: ["/internal"]},
            ],
            sitemaps: ({mode}) =>
                mode === "production"
                    ? [
                        "https://example.com/sitemap.xml",
                        "https://example.com/sitemap-news.xml",
                    ]
                    : [],
        }),
    ],
});

API

generateRobotsTxt(options ? : RobotsOptions)
:
Plugin

RobotsOptions

| Option | Type | Default | Description | |-----------------|---------------------------------|----------------------------------------------------|------------------------------------------------------------------------| | filename | string | "robots.txt" | Output filename. | | policies | RobotsPolicy[] | [{ userAgent: "*", allow: ["/"], disallow: [] }] | Explicit user-agent blocks. If provided, policyBuilder is ignored. | | policyBuilder | (ctx) => RobotsPolicy[] | — | Build policies dynamically. Receives { mode, command, root }. | | sitemaps | string[] \| (ctx) => string[] | — | Sitemaps to append as Sitemap: <url> lines. | | footerComment | string \| (ctx) => string | — | Appends a trailing comment (prefixed with #). | | noStoreInDev | boolean | true | If true, serves /robots.txt in dev with Cache-Control: no-store. |

RobotsPolicy

type RobotsPolicy = {
    userAgent?: string;   // default "*"
    allow?: string[];     // e.g., ["/"]
    disallow?: string[];  // e.g., ["/admin", "/preview"]
};

policyBuilder context

type Context = {
    mode: string;                         // "development" | "production" | custom
    command: "serve" | "build";           // Vite command
    root: string;                         // project root
};

Common recipes

1) Block everything in non‑production

generateRobotsTxt({
    policyBuilder: ({mode}) =>
        mode === "production"
            ? [{userAgent: "*", allow: ["/"], disallow: []}]
            : [{userAgent: "*", disallow: ["/"]}],
});

2) Multiple user‑agents

generateRobotsTxt({
    policies: [
        {userAgent: "Googlebot", allow: ["/"], disallow: ["/no-google"]},
        {userAgent: "Bingbot", allow: ["/"], disallow: ["/no-bing"]},
        {userAgent: "*", allow: ["/"], disallow: ["/secret"]},
    ],
});

3) Multiple sitemaps

generateRobotsTxt({
    sitemaps: [
        "https://example.com/sitemap.xml",
        "https://example.com/sitemap-news.xml",
    ],
});

4) Custom output location / file name

generateRobotsTxt({
    filename: "robots.txt"
});

How it works

• Dev (vite serve): Adds a middleware that responds to GET /robots.txt. By default the response includes Cache-Control: no-store so browsers don’t cache it while iterating.
• Build (vite build): Writes robots.txt to your static directory. The plugin tries static/ first (for SvelteKit), otherwise uses public/. If the folder doesn’t exist, it will be created.

If you already have a physical robots.txt in your static folder, the plugin will overwrite it on build. In dev, the middleware takes precedence for the /robots.txt route.

Troubleshooting

• 404 in dev → Ensure the plugin is in the plugins array (after your main framework plugin). Hit http://localhost:<port>/robots.txt directly.
• Not updating in dev → Your browser may be caching aggressively. The plugin sends no-store, but you can also hard-refresh.
• Adapter / hosting headers → Some platforms may cache static assets. If you must serve a static robots.txt with no cache, configure headers at the edge (e.g., Netlify/Vercel/Nginx).

Contributing

PRs and issues welcome! Please run npm run build before publishing/packaging.

License

MIT © 2025