Sitemind

Generate AI-facing manifests (ai.json + ai.txt) for any website. Give AI agents a structured, crawlable summary of your site's content, navigation, data, and capabilities.

Why

Websites are built for humans — interactive JavaScript, client-rendered DOM, complex navigation. AI agents navigating these sites are essentially screen-scraping: slow, brittle, lossy.

Sitemind solves this by generating a standardized pair of files that any AI agent can consume:

• ai.json — machine-readable manifest with site metadata, entities, actions, and navigation
• ai.txt — human/LLM-readable markdown summary

Think robots.txt for discoverability, sitemap.xml for structure — but purpose-built for AI agents.

Install

Go

go install github.com/griffincancode/sitemind/cmd/sitemind@latest

npm

npm install -g sitemind

From source

git clone https://github.com/griffincancode/sitemind.git
cd sitemind
make install

Usage

Scan your codebase (primary workflow)

Define a sitemind.config.yaml with your data schemas and source files:

site:
  name: "My Store"
  url: "https://mystore.com"
  description: "Premium clothing retailer"
  purpose: "e-commerce"

entities:
  product:
    fields:
      name: string
      sku: string
      price: number
      category: string
      description: string
    sources:
      - "data/products/*.json"
      - "src/data/*.ts"

  category:
    fields:
      name: string
      slug: string
    sources:
      - "data/categories.yaml"

Then run:

sitemind scan

Sitemind reads your source files (JSON, YAML, TypeScript, JavaScript, CSV), matches against your schema, and generates ai.json + ai.txt.

Initialize a config

sitemind init --name "My Site" --url "https://mysite.com" --description "A SaaS platform" --config

Crawl a live site

sitemind generate https://mysite.com --depth 3 --max-pages 100

Validate

sitemind validate ai.json

Programmatic Usage (Node.js)

const { generate, validate, init } = require("sitemind");

generate("https://mysite.com", { depth: 2, maxPages: 50 });

const result = validate("ai.json");
console.log(result.valid ? "Valid" : result.output);

ai.json Spec (v1.0)

{
  "version": "1.0",
  "site": {
    "name": "My Store",
    "url": "https://mystore.com",
    "description": "Premium clothing retailer",
    "purpose": "e-commerce",
    "language": "en",
    "lastUpdated": "2026-02-17T00:00:00Z"
  },
  "entities": {
    "product": {
      "schema": {
        "name": "string",
        "sku": "string",
        "price": "number",
        "category": "string"
      },
      "count": 23,
      "records": [
        { "name": "Waxed Canvas Jacket", "sku": "JK-001", "price": 189.99, "category": "outerwear" }
      ]
    }
  },
  "pages": [
    {
      "path": "/",
      "title": "Home",
      "description": "Landing page",
      "contentType": "landing",
      "actions": [
        {
          "type": "search",
          "description": "Search products",
          "target": "/search",
          "method": "GET",
          "parameters": [
            { "name": "q", "type": "text", "required": true }
          ]
        }
      ],
      "links": ["/about", "/pricing"],
      "priority": 1.0
    }
  ]
}

Supported Source Formats

JSON · YAML · TypeScript · JavaScript · CSV

Entity Field Types

string · number · bool · price · float · int

Content Types (pages)

landing · article · product · form · dashboard · auth · documentation · other

Action Types

navigate · search · submit · authenticate · download

CLI Reference

| Command | Description | |---------|-------------| | sitemind init | Create a starter config or ai.json | | sitemind scan | Scan codebase and generate manifests from config | | sitemind generate <url> | Crawl a live site and generate manifests | | sitemind validate [file] | Validate against the spec |

Scan Flags

| Flag | Default | Description | |------|---------|-------------| | --dir | . | Project root directory |

Generate Flags

| Flag | Default | Description | |------|---------|-------------| | --depth | 3 | Max crawl depth | | --max-pages | 50 | Max pages to crawl | | --timeout | 30 | HTTP timeout (seconds) | | --name | | Override site name | | --description | | Override description | | --purpose | | Site purpose | | --output-json | ai.json | JSON output path | | --output-txt | ai.txt | TXT output path |

Architecture

cmd/sitemind/       CLI entry point (cobra)
internal/
  spec/             Type definitions + validation
  config/           Config file loading (sitemind.config.yaml)
  scanner/          Codebase scanner (JSON, YAML, TS, JS, CSV)
  crawler/          Concurrent site crawler (for live URLs)
  generator/        Produces ai.json + ai.txt
npm/                npm package wrapper (downloads Go binary)

License

MIT