@lacspace/robots

Build & parse robots.txt — with AI-crawler block presets and Next.js output.

Typed per-user-agent rules, Sitemap: / Host: / Crawl-delay:, a parser, Next.js robots.ts output — and a one-liner to block AI crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot…).

• 🤖 robots() builder · parseRobots() parser
• 🚫 blockAiBots() + the AI_BOTS list (18 known crawlers)
• ▲ toNextRobots() for app/robots.ts
• ⚡ Zero dependencies · 🌍 isomorphic · 📦 ESM + CJS · fully typed

Install

npm install @lacspace/robots      # or pnpm add / yarn add / bun add

Build robots.txt

import { robots } from "@lacspace/robots";

robots({
  groups: [
    { userAgent: "*", disallow: ["/admin", "/api"], allow: ["/api/public"] },
    { userAgent: "Googlebot", disallow: [] }, // allow all
  ],
  sitemap: "https://lacspace.com/sitemap.xml",
  host: "lacspace.com",
});

Block AI crawlers, allow everyone else

import { blockAiBots } from "@lacspace/robots";

blockAiBots({ sitemap: "https://lacspace.com/sitemap.xml" });
// User-agent: *
// Disallow:
//
// User-agent: GPTBot
// User-agent: ClaudeBot
// User-agent: CCBot
// … (Disallow: / for each)
//
// Sitemap: https://lacspace.com/sitemap.xml

Next.js app/robots.ts

import { toNextRobots } from "@lacspace/robots";

export default function robots() {
  return toNextRobots({
    groups: [{ userAgent: "*", allow: ["/"], disallow: ["/admin"] }],
    sitemap: "https://lacspace.com/sitemap.xml",
  });
}

Parse an existing file

import { parseRobots } from "@lacspace/robots";

const { groups, sitemaps, host } = parseRobots(txt);

The Lacspace SEO Kit

| Package | For | | --- | --- | | @lacspace/seo | Metadata & JSON-LD | | @lacspace/sitemap | sitemap.xml | | @lacspace/robots | robots.txt (this package) | | @lacspace/llms-txt | llms.txt / llms-full.txt | | @lacspace/site-verify | Search-engine verification | | @lacspace/rss | RSS / Atom / JSON feeds | | @lacspace/slugify | SEO URL slugs |

New in 1.2 — stack presets & a crawlability matcher

import { nextjsRobots, blockAll, envRobots, allowSearchBlockTraining, isAllowed, parseRobots } from "@lacspace/robots";

// Stack-aware defaults (Next.js/_next, WordPress, Shopify)
export default () => nextjsRobots({ sitemap: "https://x.com/sitemap.xml" });

// Block everything on preview/staging, index in production
envRobots(process.env.VERCEL_ENV === "production", { sitemap });

// Allow search + answer engines but block AI-training crawlers
allowSearchBlockTraining({ sitemap });

// Test whether a URL is crawlable (longest-match wins; ties favour Allow)
const parsed = parseRobots(txt);
isAllowed("/admin/secret", parsed);        // false
isAllowed("/files/a.pdf", parseRobots("User-agent: *\nDisallow: /*.pdf$")); // false

New in 1.2 — one-line robots.txt from your site URL

import { robotsForSite } from "@lacspace/robots";

robotsForSite({ url: "https://acme.com" }, { blockAi: true });
// allow all · Sitemap: https://acme.com/sitemap.xml · Host: … · blocks GPTBot, ClaudeBot, CCBot, Google-Extended…

Pairs with defineSite() from @lacspace/seo — robotsForSite(site.config, { blockAi: true }).

Licensing

This package is free under the Lacspace Free Licence — MIT-equivalent freedoms. Use it in personal and commercial projects at no cost; just keep the notice.

Not every Lacspace package is free. We also offer Commercial (paid), Client-specific, and Private (proprietary) packages under separate terms. See the full Lacspace Licence Centre.

Part of the Lacspace ecosystem — 35 zero-dependency, isomorphic TypeScript packages.

All packages ↗ · npm org ↗ · Licence Centre ↗ · GitHub ↗