doks-core

Framework engine for doks, an open-source RAG-optimized documentation site on Next.js + MDX. Bring your own keys, no SaaS layer, no telemetry.

This package ships:

• React components (<DocsShell>, <Header>, <LeftSidebar>, <Spotlight>, <RightRail>)
• MDX components (<Hero>, <Callout>, <QJump>, <Steps>, <Tabs>, <Figure>, …)
• The page renderer (DocPage for /docs/[[...slug]]), the search route factory, the 404 / error boundaries
• A pluggable VectorStore contract with three adapters out of the box: SQLite (default), D1 (in-Worker), D1 over the Cloudflare REST API (for Node-side ingest)
• An ingest script that walks content/docs/, embeds chunks, and writes to whichever store is configured
• A build-time MDX content snapshot generator (doks build:content) that lets doc pages render at request time on edge runtimes without filesystem reads
• A withDoks(nextConfig) wrapper for next.config.mjs that runs the snapshot generator at config-load time
• The doks CLI: upgrade, ensure-index, build:content, d1:init, setup-cloudflare, deploy:cloudflare

Quick start

The fast path: don't install this directly, scaffold a project that uses it.

npx create-doks my-docs
cd my-docs
npm run ingest    # builds data/docs.db
npm run dev       # http://localhost:3000

That gives you a working Next.js + MDX docs site with doks-core already wired up.

Manual install

If you're grafting doks onto an existing Next.js app:

npm install doks-core

Then add it to next.config.mjs (the withDoks wrapper runs the content-snapshot generator at config load):

import { withDoks } from 'doks-core/next';

const nextConfig = {
  transpilePackages: ['doks-core'],
  serverExternalPackages: ['better-sqlite3', 'sqlite-vec'],
};

export default withDoks(nextConfig);

Import the styles from your global stylesheet:

/* app/globals.css */
@import 'doks-core/styles.css';

Pick a vector-store adapter in lib/doks.config.ts:

// lib/doks.config.ts
import { createSqliteStore } from 'doks-core/adapters/sqlite';
import type { VectorStore } from 'doks-core';

export const vectorStore: VectorStore = createSqliteStore({
  path: 'data/docs.db',
});

Wire the routes:

// app/docs/[[...slug]]/page.tsx
import { DocPage } from 'doks-core';
export { generateStaticParams, generateMetadata } from 'doks-core';
export default DocPage;

Register the build-time content snapshot in your root layout (side-effect import; the file is generated by doks build:content):

// app/layout.tsx
import 'doks-core/styles.css';
import '@/lib/doks-content.gen';
// …rest of layout

Add the generator to your scripts so it runs before next dev and next build:

{
  "scripts": {
    "predev": "doks build:content && doks ensure-index",
    "prebuild": "doks build:content"
  }
}

// app/api/docs/search/route.ts
import { createSearchHandler } from 'doks-core';
import { vectorStore } from '@/lib/doks.config';

export const runtime = 'nodejs';
export const dynamic = 'force-dynamic';
export const { POST, GET } = createSearchHandler(vectorStore);

Run the ingest script to build the index:

npm pkg set scripts.ingest='tsx node_modules/doks-core/dist/scripts/ingest.js'
npm run ingest

For a complete walkthrough see the install docs.

CLI

The package installs a doks bin:

npx doks upgrade                  # bump doks-core, run pending migrations
npx doks upgrade --dry-run        # preview without applying
npx doks ensure-index             # build data/docs.db if missing (predev hook)
npx doks build:content            # generate lib/doks-content.gen.ts (snapshot)
npx doks d1:init                  # print the D1 chunks-table schema (stdout)
npx doks d1:init --remote <db>    # run the schema against a remote D1
npx doks d1:init --local <db>     # run against a local D1
npx doks setup-cloudflare         # provision D1 + R2 + schema, print wrangler.jsonc
npx doks deploy:cloudflare        # end-to-end: peer deps, provision, mutate
                                  # configs, prompt for token. --dry-run supported.
npx doks --help

setup-cloudflare accepts --worker, --db, --bucket overrides and --skip-d1, --skip-r2, --skip-schema to provision a subset.

Run from inside a project that has doks-core as a dependency.

The scaffold wires predev: doks ensure-index so the first npm run dev builds the SQLite index automatically. ensure-index quietly noops for D1 setups; you stay in control of when to ingest.

Requirements

| Tool | Version | Notes | | --- | --- | --- | | Node.js | >= 20 | Next 15 requires ^18.18 \|\| ^19.8 \|\| >=20. doks targets 20. | | npm | bundled | Any recent version with workspace support. | | C toolchain | platform-specific | Only needed for the SQLite adapter (better-sqlite3 ships native bindings). macOS: xcode-select --install. Debian/Ubuntu: sudo apt-get install -y build-essential. Windows: use WSL. D1-only deployments don't need a C toolchain. | | wrangler, @opennextjs/cloudflare, @cloudflare/workers-types | latest | Optional. Only needed for Cloudflare D1 deployments. Listed as optional peer deps. |

What's in the tarball

dist/             compiled JS + .d.ts (entry: dist/index.js)
src/styles/       CSS bundles, imported via 'doks-core/styles.css'
bin/doks.js       the `doks` CLI
migrations/       one-shot scripts run by `doks upgrade`
LICENSE
README.md

Public API

Everything exported from dist/index.d.ts is the stable surface. Anything not exported there is internal and may change between versions.

Subpath exports keep heavy or platform-specific code out of consumer bundles (notably: the SQLite adapter pulls better-sqlite3, so it lives under a subpath and never ships to edge runtimes through the top-level barrel):

| Subpath | Use for | | --- | --- | | doks-core | Top-level barrel: components, types, createSearchHandler, VectorStore | | doks-core/styles.css | Framework CSS (@import from your globals.css) | | doks-core/adapters/sqlite | createSqliteStore({ path }). Node only. | | doks-core/adapters/d1 | createD1Store(D1Database). Edge-safe; pair with getCloudflareContext(). | | doks-core/adapters/d1/http | createD1HttpStore({ accountId, databaseId, apiToken }). Node-side ingest into remote D1 over the Cloudflare REST API. | | doks-core/cloudflare/open-next | Default OpenNext config that wires the R2 incremental cache. Re-export from open-next.config.ts. | | doks-core/next | withDoks(nextConfig) wrapper. Runs the content-snapshot generator at config-load time. | | doks-core/runtime/content | setContentMap, getBundledMap — cache used by the data layer. The generated lib/doks-content.gen.ts calls setContentMap. | | doks-core/scripts/buildContent | Programmatic API for the snapshot generator (typically invoked via the doks build:content CLI). | | doks-core/api/search | Pre-built search route handler (legacy; prefer createSearchHandler) | | doks-core/not-found | The branded 404 page | | doks-core/error | Runtime error boundary (use as app/error.tsx) | | doks-core/global-error | Root-layout-crash boundary (use as app/global-error.tsx) | | doks-core/scripts/ingest | The ingest script (tsx node_modules/doks-core/dist/scripts/ingest.js) | | doks-core/mdx-components | Pre-registered MDX components map |

Bundled-content runtime (0.3.0+)

doks build:content walks content/docs/, captures every MDX file's frontmatter + raw source + slug + href into a TypeScript module (lib/doks-content.gen.ts), and registers it with the runtime cache via a side-effect import in your root layout. After registration:

• getDocBySlug, getAllDocs, getDocSource, and buildDocTree resolve from memory.
• The shipped DocPage no longer reads content/docs/ at request time.

This is what makes it possible to serve doc pages from edge runtimes (Cloudflare Workers, Vercel Edge) — they have no filesystem.

The data layer falls back to fs.readFileSync when the cache is empty, so existing 0.2.x consumers keep working without changes. Migrate with npx doks upgrade, which adds the predev/prebuild hooks, the side-effect import in app/layout.tsx, and the withDoks wrapper in next.config.mjs.

Cloudflare deployment (D1 + R2)

doks-core has first-class support for Cloudflare Workers via D1 (vectors) and R2 (OpenNext incremental cache). One command does everything:

npx wrangler login
npx doks deploy:cloudflare
DOKS_CONFIG=lib/doks.config.ingest.ts npm run ingest
npm run deploy

deploy:cloudflare installs peer deps, provisions D1 + R2, runs the chunks schema, writes wrangler.jsonc, swaps lib/doks.config.ts to the D1 thunk, writes lib/doks.config.ingest.ts, uncomments the OpenNext dev hook in next.config.mjs, and prompts for the API token to populate .env.local. Pass --dry-run to preview.

If you scaffolded with npx create-doks and picked the Cloudflare target at the prompt, the configs are already wired — only the cloud provisioning step remains.

Full walkthrough (incl. the granular manual path): Deployment guide.

Versioning

Semver. Public API breaks bump the minor pre-1.0 (or major post-1.0) and ship with a migration script under migrations/<target-version>.js that doks upgrade runs automatically. See migrations/README.md for the policy if you're contributing a breaking change.

Links

• Full docs: https://github.com/getdoks/doks
• Issues: https://github.com/getdoks/doks/issues
• Scaffold CLI: create-doks

License

MIT. See LICENSE.