@iandolan/astro-sitemap

The crawl-control layer around your sitemap: robots.txt with an explicit AI crawler policy, llms.txt, and IndexNow submission. @astrojs/sitemap already emits the XML, so this covers what sits around it.

import { buildRobots, buildLlmsTxt, submitToIndexNow } from '@iandolan/astro-sitemap';

buildRobots({ site, disallow: ['/thank-you'], ai: 'block-training' });
buildLlmsTxt({ site, name: 'Dog Tail', summary: '...', sections });

The AI crawler policy splits bots by what blocking one costs you. training bots (GPTBot, Google-Extended, ClaudeBot) take your copy into a training set. search bots (OAI-SearchBot, PerplexityBot) are how you get cited in answers. Blocking the second group removes you from AI results, which for most local and professional-services businesses is lost visibility rather than protected IP. ai: 'block-training' is the middle setting.

IndexNow covers Bing and Yandex. Google does not participate, so it complements a sitemap rather than replacing it.

seokit-indexnow --site https://example.com --key $INDEXNOW_KEY --sitemap dist/sitemap-index.xml --dry-run

Part of the Dolan SEO Suite, WordPress SEO plugin functionality rebuilt for static Astro sites. Plain ESM, no build step.

MIT © Ian Dolan