thorbit-knowledge-graph

Build canonical entity libraries from any website. Point it at a site (or a list of pages) and it extracts and links every salient entity to its canonical IDs — Wikidata QID, Wikipedia, DBpedia, productontology, and the Google/Freebase /m/ mid — then dedupes, cleans, categorizes, and ranks them into a tidy library, ready for a final AI cleanup pass.

One repo, three surfaces over one engine (src/engine/):

• CLI (bin/cli.mjs) — run it locally, zero-config.
• MCP server (mcp/) — expose it to Claude; runs the engine locally or calls the hosted API.
• Vercel API (api/) — host it on thorbit.ai with server-side keys, gated by approved API keys.

Entity linking is content-grounded (via TextRazor), so it resolves terms in context — "rehab" becomes Drug rehabilitation, not a band's album.

CLI

npm install
cp .env.example .env            # paste one or more free TextRazor keys (textrazor.com)

node bin/cli.mjs run --url https://example-rehab.com --niche drug-rehab --out ./out
node bin/cli.mjs run --urls "https://a.com/p1,https://a.com/p2" --niche default
node bin/cli.mjs clean --raw ./out/raw.json --niche drug-rehab
node bin/cli.mjs merge --into ./rehab-seed.json --from ./out/library.json --label example-rehab.com

Outputs in out/: raw.json (everything), library.json (deduped, cleaned, ranked), library-core.json (high-confidence subset), geo.json (locations split out), summary.json. Niches: drug-rehab, roofing, default — add your own in src/engine/niches.mjs.

If installed (npm i -g . or via npm), the commands are thorbit-kg run ....

MCP

Tools: kg_build_library (scraped pages / site / url-list → library; lean result + saved reference), kg_emit_schema (one page → finished schema.org JSON-LD), kg_emit_schema_bulk (many pages → JSON-LD, in parallel), the saved-KG catalog (kg_library_save / _list / _get / _approve / _remove), kg_resolve_term (one phrase → IDs, no crawl), kg_merge_seed (accumulate a per-niche seed). Results are lean by design — the full library is saved and referenced by path, never dumped into context. The other bucket in each result is the deliberate hand-off for the model to finish.

Saved-KG catalog (build once, reuse by name)

A named, browsable catalog of libraries so you don't rebuild every time. Build → save → approve → reuse:

kg_build_library(...)                              → libraryUrl
kg_library_save({ name: "acme", libraryUrl })      → saved as PENDING
kg_library_list()                                  → approved only (includePending:true to review the queue)
kg_library_get({ name: "acme" })                   → inspect the entities before approving
kg_library_approve({ name: "acme" })               → now usable
kg_emit_schema({ pageType: "home", libraryName: "acme", business })

The approval gate is a security boundary, not a formality. A scraped library is untrusted content — an entity name or description could carry a prompt-injection payload that reaches generated schema. So a saved KG is unusable until a human approves it: kg_emit_schema/_bulk refuse an unapproved libraryName (403), the model is instructed to treat library content as data (never instructions), and it must never self-approve. Builds are not auto-registered — you choose what enters the catalog. Catalog + libraries persist in Vercel Blob.

Emit schema (kg_emit_schema)

Turns one page into finished schema.org JSON-LD for its type (home / service / about / blog). The split is the point: the knowsAbout/about/mentions DefinedTerm blocks are linked from the grounded library — every @id, name, and 5-link sameAs (Wikidata, Wikipedia, DBpedia, productontology, Google/Freebase mid) is copied verbatim and locked, never invented — while the prose (per-term descriptions, audience pain points, serviceOutput, teaches, positioning) is written through OpenRouter by the exact google/gemini-3.7-flash model at high reasoning effort. After generation, every DefinedTerm is re-locked to the library's canonical IDs and any entity the model invented is dropped — so a hallucinated sameAs can't ship.

kg_emit_schema({
  pageType: "service",
  content: "<the page text>",          // builds the library, or pass library / libraryUrl from kg_build_library
  business: { name, url, telephone, address, geo, areaServed, aggregateRating }  // used verbatim, never fabricated
})  →  { jsonld, report:{ entitiesUsed, droppedHallucinated, model } }

The Knowledge Graph surface owns this generation policy. Its internal model adapter and hosted emit endpoint use explicit accepted-input contracts, so aliases such as modelId, reasoning_effort, thinking_budget, models, transforms, and every other unrecognized top-level field fail before a provider request. A noncanonical OPENROUTER_MODEL or legacy OPEN_ROUTER_MODEL setting also fails before inference. OpenRouter fallback routing is disabled, providers must support the requested parameters, and the response must report the exact requested model. The returned report.model is that verified provider receipt, not a configured label. Requests use JSON-object response mode and do not send temperature, top_p, top_k, or candidate_count. TextRazor extraction and canonical entity linking remain deterministic and outside the LLM path.

After the model returns, every DefinedTerm is re-locked to the library's canonical IDs, invented entities are dropped, and a geo guard strips any areaServed sameAs that doesn't match a business.areaServed you supplied (so a wrong place ID can't ship either). Needs OPENROUTER_API_KEY (local mode) or it's server-side in hosted mode. Templates live in src/templates/.

Bulk — kg_emit_schema_bulk takes a pages array (each with its own pageType + content) plus a shared business and optional shared library/libraryUrl, and emits them in parallel (bounded concurrency, default 3). Each page is a separate model call, so the per-page timeout holds and one slow page never blocks the rest. Lean by default (save: true) — returns a schemaUrl/savedTo + report per page; set save: false for full inline JSON-LD.

kg_emit_schema_bulk({
  business: { name, url, telephone, areaServed },
  libraryUrl: "<from kg_build_library>",
  pages: [
    { pageType: "home",    content: "..." },
    { pageType: "service", content: "..." },
    { pageType: "about",   content: "..." }
  ]
})  →  { count, ok, failed, results: [{ pageType, schemaUrl, report }] }

Pair it with a scraper (the intended flow)

This server is the entity-linking half. A scraper gathers the page content (handling JS rendering, blocks, and full-site crawls a plain fetch can't), then you hand that content to kg_build_library — it links every entity to Wikidata/Wikipedia/DBpedia/Freebase, dedupes, cleans, and ranks. With MCP Scraper:

1. extract_url("https://example.com")            → scraped page content (HTML or markdown)
2. kg_build_library({ pages: [{ url, content }], niche: "default" })
                                                  → linked, cleaned entity library

content can be HTML, markdown, or plain text — it's submitted to TextRazor with the right cleanup mode automatically (pass format to be explicit). Fallback (no scraper): give kg_build_library a url or urls and it self-fetches with a built-in plain-HTTP crawler (no JS rendering).

Register with Claude Code / Desktop:

{
  "mcpServers": {
    "thorbit-knowledge-graph": {
      "command": "npx",
      "args": ["-y", "thorbit-knowledge-graph"]
    }
  }
}

• Local mode: put TextRazor keys in the MCP's environment; it runs the engine itself.
• Hosted mode: set THORBIT_KG_SERVER_URL + THORBIT_KG_API_KEY and it calls your Vercel API instead — friends need no keys of their own.

Verify: npm test, npm run smoke, npm run test:mcp (real stdio handshake plus a deterministic local tool fixture that fails on MCP errors).

Vercel API (hosted)

Endpoints: POST /api/kg/build, POST /api/kg/emit, POST /api/kg/resolve, POST /api/kg/registry, GET /api/kg/library/:id. TextRazor and OpenRouter keys live server-side; results store in Vercel Blob. The emit endpoint accepts only pageType, content, library, libraryUrl, libraryName, business, niche, and save; every other top-level field is rejected with 400.

Auth (now): approved keys, no database, no billing. Mint one per person and add it to KG_API_KEYS:

node scripts/genkey.mjs alice      # -> kg_alice_…   give this to Alice

Every request needs x-api-key: <approved key>. Revoke = remove from KG_API_KEYS + redeploy. (Later this swaps for the thorbit-subscription model, same endpoints.)

Deploy:

vercel link
vercel env add TEXTRAZOR_API_KEY      # repeat _2.._N for your rotation pool
vercel env add KG_API_KEYS            # comma-separated minted keys
vercel env add BLOB_READ_WRITE_TOKEN  # Vercel Blob
vercel env add BLOB_PUBLIC_BASE       # https://<store>.public.blob.vercel-storage.com
vercel deploy --prod

api/kg/build is maxDuration: 300; on Hobby plan cap pages with max≈20.

How it cleans

Dedupe by wikidataId → drop universal boilerplate (cookie/GDPR/analytics/marketing) + generic bare nouns → split geo → categorize by knowledge-graph type + niche keywords → rank by relevance × page-spread × domain-fit. It never invents IDs — no Freebase mid means googleKg/freebaseId are null, so you never ship a sameAs you can't verify.

Cost

TextRazor free tier is 500 requests/day per key (rotated); a build is ~1 call/page. kg_resolve_term hits only the free Wikidata API.

MIT licensed.