@carrot-foundation/ai-discovery

AI discovery primitives for the Carrot Network: Schema.org JSON-LD builders, llms.txt generators, AI-aware robots.txt helpers, Markdown mirror routes, read-only MCP scaffolding, keyword search, IndexNow submission helpers, and a public docs glossary client.

Use this package when a public Carrot site or compatible ecosystem tool needs to expose the same content clearly to search engines, AI crawlers, LLM clients, and human developers.

Status

Release-triggering merges to main publish automatically after CI succeeds, so only merge them after Cristiano has reviewed the package PR and tarball.

Installation

pnpm add @carrot-foundation/ai-discovery
# or
npm install @carrot-foundation/ai-discovery

Requires Node.js >=22. The package is ESM-first and ships CommonJS dual output for consumers that still use require().

What It Provides

| Module | Import path | Purpose | | --------------- | -------------------------------------------- | --------------------------------------------------------------- | | JSON-LD | @carrot-foundation/ai-discovery/jsonld | Build validated Schema.org graph nodes for public pages. | | Robots | @carrot-foundation/ai-discovery/robots | Generate robots.txt policies for known AI bots. | | llms.txt | @carrot-foundation/ai-discovery/llms-txt | Generate /llms.txt and /llms-full.txt style Markdown files. | | Markdown mirror | @carrot-foundation/ai-discovery/md-mirror | Create a Request -> Response Markdown route handler. | | MCP server | @carrot-foundation/ai-discovery/mcp-server | Register read-only tools and handle Streamable HTTP lifecycle. | | Orama search | @carrot-foundation/ai-discovery/orama | Build, search, serialize, and restore keyword-only indexes. | | IndexNow | @carrot-foundation/ai-discovery/index-now | Normalize, chunk, submit, and summarize IndexNow URL batches. | | Glossary client | @carrot-foundation/ai-discovery/glossary | Query the public Carrot Docs MCP glossary tool with cache. | | Root namespace | @carrot-foundation/ai-discovery | Namespaced exports for all modules. |

Quick Start

Add JSON-LD to a page

import {
  article,
  compose,
  organization,
  website,
} from "@carrot-foundation/ai-discovery/jsonld";

const graph = compose([
  organization({
    name: "Carrot Foundation",
    url: "https://carrot.eco",
    logo: "https://carrot.eco/logo.png",
    description: "Public infrastructure for environmental assets.",
  }),
  website({
    name: "Carrot Documentation",
    url: "https://docs.carrot.eco",
    inLanguage: ["en", "pt-BR"],
    searchUrlTemplate: "https://docs.carrot.eco/search?q={query}",
  }),
  article({
    url: "https://docs.carrot.eco/docs/concepts/dmrv",
    headline: "Digital Measurement, Reporting and Verification",
    description:
      "How Carrot structures source data, evidence, and verification.",
    datePublished: "2026-04-15",
    dateModified: "2026-05-01",
    author: {
      name: "Carrot Foundation",
      sameAs: ["https://carrot.eco"],
    },
    inLanguage: "en",
    image: "https://docs.carrot.eco/og/docs/concepts/dmrv.png",
  }),
]);

const jsonLd = JSON.stringify(graph);

Render jsonLd in an application/ld+json script tag. Builders validate input with Zod before returning plain objects, so invalid URLs, dates, language tags, or missing required fields fail early.

Generate an AI-aware robots.txt

import { buildRobots } from "@carrot-foundation/ai-discovery/robots";

export function GET() {
  return new Response(
    buildRobots({
      allow: "fetchers-only",
      sitemap: "https://docs.carrot.eco/sitemap.xml",
      disallowPaths: ["/api/", "/internal/"],
    }),
    {
      headers: {
        "content-type": "text/plain; charset=utf-8",
      },
    },
  );
}

allow supports three modes:

| Mode | Behavior | | --------------- | ------------------------------------------------------------ | | all-ai-bots | Allows known AI training, search-index, and user-fetch bots. | | fetchers-only | Allows search and user-fetch bots, blocks training bots. | | block-all-ai | Blocks all known AI bots listed by this package. |

The canonical bot list lives in AI_BOTS, with each entry classified as training, search-index, or user-fetch.

Build llms.txt files

import {
  buildFull,
  buildIndex,
} from "@carrot-foundation/ai-discovery/llms-txt";

const index = buildIndex({
  site: {
    name: "Carrot Documentation",
    url: "https://docs.carrot.eco",
    tagline: "The public reference for the Carrot Network.",
  },
  sections: [
    {
      title: "Concepts",
      urls: [
        {
          title: "dMRV",
          url: "https://docs.carrot.eco/docs/concepts/dmrv",
          description:
            "Digital Measurement, Reporting and Verification in Carrot.",
        },
      ],
    },
  ],
});

const full = buildFull({
  pages: [
    {
      title: "dMRV",
      url: "https://docs.carrot.eco/docs/concepts/dmrv",
      markdown: "# dMRV\n\nDigital Measurement, Reporting and Verification.",
    },
  ],
});

buildIndex() escapes Markdown-sensitive text and formats URLs for a compact index file. buildFull() concatenates page Markdown with source URL comments so LLM clients can trace each block back to its canonical page.

Expose a Markdown mirror route

import { createMdMirrorRoute } from "@carrot-foundation/ai-discovery/md-mirror";

export const GET = createMdMirrorRoute<{ slug?: string[] }>({
  async load({ params }) {
    const slug = params.slug?.join("/") ?? "index";
    const page = await loadPageAsMarkdown(slug);

    if (page === null) return null;

    return {
      markdown: page.markdown,
      sourceUrl: `https://docs.carrot.eco/docs/${slug}`,
    };
  },
});

The route helper returns 404 when load() returns null, otherwise it serves text/markdown; charset=utf-8, sets x-source-url, and applies a cache-control header. The default cache policy is public, max-age=300, s-maxage=86400.

Register read-only MCP tools

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";
import { registerReadOnlyTool } from "@carrot-foundation/ai-discovery/mcp-server";

const server = new McpServer({ name: "carrot-docs", version: "1.0.0" });

registerReadOnlyTool(server, {
  name: "search_docs",
  title: "Search docs",
  description: "Search public Carrot documentation.",
  inputSchema: {
    query: z.string(),
  },
  handler: (input) => ({
    results: searchPublicDocs(input.query),
  }),
});

registerReadOnlyTool() applies the same safe annotations to each tool: readOnlyHint: true, destructiveHint: false, idempotentHint: true, and openWorldHint: false. createMcpHttpHandler() provides a generic Streamable HTTP request lifecycle with app-provided rate keys, host-gated telemetry, and tool-name allowlists so raw bodies, raw IP-derived keys, headers, and prompt text are not emitted by the library.

Build a keyword search index

import {
  buildSearchIndex,
  searchIndex,
  serializeSearchIndex,
} from "@carrot-foundation/ai-discovery/orama";

const index = await buildSearchIndex([
  {
    id: "en/blog/massid",
    title: "MassID launch",
    url: "https://www.carrot.eco/en/blog/massid",
    text: "MassID traceability for circular economy data.",
    locale: "en",
    kind: "post",
  },
]);

const results = await searchIndex(index, { query: "massid", limit: 5 });
const serialized = serializeSearchIndex(index);

The Orama wrapper is keyword-only in 1.1.0. Embeddings and cross-property federation are intentionally left to later phases.

Submit IndexNow URL batches

import {
  chunkIndexNowUrls,
  formatGitHubActionsSummary,
  submitIndexNowUrls,
} from "@carrot-foundation/ai-discovery/index-now";

const urls = [
  "https://www.carrot.eco/en",
  "https://www.carrot.eco/en/blog/massid",
];

const indexNowKey = process.env.INDEXNOW_KEY;
if (!indexNowKey) {
  throw new Error("INDEXNOW_KEY is required");
}

for (const batch of chunkIndexNowUrls(urls)) {
  const result = await submitIndexNowUrls({
    host: "www.carrot.eco",
    key: indexNowKey,
    keyLocation: "https://www.carrot.eco/indexnow-key.txt",
    urls: batch,
    fetch,
  });

  console.log(formatGitHubActionsSummary(result));
}

normalizeIndexNowUrls() keeps only unique HTTPS URLs under the configured host. Non-2xx responses return typed failures without including raw response bodies. Set INDEXNOW_KEY before calling submitIndexNowUrls().

Query the public docs glossary MCP

import {
  createDocsGlossaryClient,
  createMemoryGlossaryCache,
  getGlossaryTerm,
} from "@carrot-foundation/ai-discovery/glossary";

const glossary = createDocsGlossaryClient({
  endpoint: "https://docs.carrot.eco/mcp",
  fetch,
  cache: createMemoryGlossaryCache({ ttlMs: 60_000 }),
});

const trc = await getGlossaryTerm(glossary, { term: "TRC", locale: "en" });

The glossary client is scoped to the public get_glossary_term tool. It sends no Authorization header by default and caches normalized term-locale lookups when a cache is provided.

JSON-LD Builders

The JSON-LD module exports builders for the public page types Carrot sites use most often:

| Builder | Schema.org node | | ------------------ | ----------------------------------------------------------- | | apiReference() | APIReference with a WebAPI target. | | article() | BlogPosting, Article, TechArticle, or HowTo. | | breadcrumbList() | BreadcrumbList. | | definedTerm() | DefinedTerm. | | definedTermSet() | DefinedTermSet. | | faqPage() | FAQPage. | | organization() | Organization. | | person() | Person. | | podcastEpisode() | PodcastEpisode, with optional media and platform links. | | podcastSeries() | PodcastSeries, with optional web feed and platform links. | | speakable() | SpeakableSpecification. | | website() | WebSite, with optional SearchAction. |

Use compose(nodes) for the canonical output shape:

{
  "@context": "https://schema.org",
  "@graph": []
}

Use schemaId(siteOrigin, kind, payload?) when you need deterministic @id values for custom nodes.

Design Principles

• Builders are pure. They return strings or plain objects and do not perform I/O.
• Validation happens at the boundary through Zod schemas.
• Optional fields are omitted when absent instead of being set to undefined.
• AI bot policy is explicit and generated from a reviewed list.
• Markdown outputs preserve canonical source URLs for traceability.
• Each top-level module is available as a subpath export for smaller consumer bundles.

Project Structure

src/
  index.ts          # Root namespaced exports
  jsonld/           # Schema.org JSON-LD builders and validation helpers
  robots/           # AI bot registry and robots.txt builder
  llms-txt/         # llms.txt index and full-content builders
  md-mirror/        # Request -> Response Markdown route helper
  mcp-server/       # Read-only MCP registration and HTTP lifecycle helpers
  orama/            # Keyword search index helpers
  index-now/        # IndexNow URL normalization and submission helpers
  glossary/         # Carrot Docs glossary MCP client
test/
  jsonld/
  robots/
  llms-txt/
  md-mirror/
  mcp-server/
  orama/
  index-now/
  glossary/

When adding a new top-level module, keep the public surface in sync:

1. Add import and require pairs to package.json exports.
2. Add the module entry to tsup.config.ts.
3. Add a namespaced re-export to src/index.ts.
4. Mirror tests under test/<module>/.

Development

pnpm install --frozen-lockfile
pnpm test
pnpm build
pnpm type-check
pnpm lint
pnpm format:check
pnpm spell-check
pnpm check

pnpm check is the full local gate and mirrors CI: format, lint, type-check, spell-check, tests, build, exported type checks, dry-run packing, and production audit.

Useful focused commands:

pnpm vitest run test/jsonld/article.test.ts
pnpm vitest -t "robots"
pnpm check-types-exports
pnpm pack --dry-run

Release

Releases are automated with semantic-release from main after the Check Code workflow succeeds for a push. Manual workflow_dispatch is kept as a recovery path.

Do not bump version manually. It stays 0.0.0-development; semantic-release determines the published version from Conventional Commits.

Related Carrot Projects

• @carrot-foundation/schemas provides Zod validation schemas and generated JSON Schemas for Carrot metadata.
• carrot-foundation/methodology-rules provides open-source rule processors for the Carrot dMRV pipeline.
• Carrot Documentation is the public reference for Carrot concepts, methodologies, and integration guides.

License

Apache License 2.0. See NOTICE for attribution, trademark, and warranty notes.