@chinesemachen/growthks-server-ts
v1.5.6
Published
GrowthKS Server - TypeScript Edition
Readme
GrowthKS Server
A self-hosted, LLM-powered knowledge base. Drop documents into a folder, let the LLM build a structured wiki, then chat — answers cite the wiki and (optionally) reach into your source code via Copilot CLI.
Highlights
- One Main Agent, one Knowledge Base — no agent management UI to worry about; the system seeds the Main Agent on first launch
- Pick the KB Directory once, init runs
wiki init— yourraw/andwiki/folders live wherever you point them - Drop files into
raw/→ click Ingest — re-runwiki ingestanytime; output goes to the structuredwiki/tree - Optional code scan — attach local repo paths or git URLs; the planner uses them for
scan_sourceactions, no indexing required - Streamed answers — Server-Sent Events for live status, planning steps, and token streaming
- Wiki-grounded Q&A — MiniSearch (CJK-aware) finds relevant pages; the LLM answers from them, with optional report export
- Direct LLM fallback — one click sends your last question to the LLM unmediated
How it works
┌─────────────────────────────────────────────┐
│ User opens http://localhost:5001 │
└─────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────┐
│ Setup panel — first run │
│ • Confirm KB Directory (default = DATA_DIR) │
│ • (Optional) Add Project Repo/Wiki paths │
│ • Click Done → `wiki init` runs in background │
└──────────────────────────────────────────────────┘
│ KB ready
▼
┌──────────────────────────────────────────────────┐
│ Chat │
│ planner → scan_source (raw/ materials + repos)│
│ → search_wiki (MiniSearch + LLM) │
│ → direct_reply (LLM only) │
│ → clarify (ask for more detail) │
└──────────────────────────────────────────────────┘
│ any time
▼
┌──────────────────────────────────────────────────┐
│ Configure panel │
│ • Add / rename / delete / edit repo paths │
│ (every edit syncs to DB immediately) │
│ • Drop new files into raw/ → click Ingest │
└──────────────────────────────────────────────────┘The KB layout under whatever directory you picked:
<KB Directory>/main/
├── raw/ # Your source materials (.md, assets, ...)
└── wiki/ # LLM-generated structured pages
├── index.md
├── overview.md
├── conventions.md
├── log.md
├── entities/
├── concepts/
├── sources/
└── analyses/Quick start (users)
1. Prerequisites
- Node.js ≥ 18 — https://nodejs.org
- GitHub CLI (
gh) — provides the credential GrowthKS uses to call GitHub Models. See Set up the LLM credential (GitHub CLI) below. - (Optional) GitHub Copilot CLI — the standalone
copilotbinary (NOT thegh-copilotgh-extension). Used for thescan_sourceagent and, when no API credential is configured, as the no-API-key reasoning fallback (a residentcopilot --acpprocess; no embeddings, so vector search degrades to lexical/MiniSearch).- Standalone download: https://aka.ms/copilot-cli
- Verify:
node --versionshould succeed.
1.5. Set up the LLM credential (GitHub CLI)
GrowthKS calls GitHub Models for its LLM. It looks for a token in this order:
GITHUB_TOKEN → GH_TOKEN → the token from gh auth token. The easiest way is to log in
with the GitHub CLI.
Step 1 — Install GitHub CLI
- Windows (winget):
winget install --id GitHub.cli - Windows (installer): https://github.com/cli/cli/releases (download the
.msi) - macOS:
brew install gh - Linux: see https://github.com/cli/cli#installation
Step 2 — Log in
🔑 Before logging in: make sure you have a Microsoft GitHub EMU account with GitHub Models access. Visit https://copilot.github.microsoft.com/ first to apply for or sign in to your Microsoft EMU account on GitHub, then use that account in the
gh auth loginstep below.
gh auth loginChoose GitHub.com → HTTPS → authenticate in the browser. Then verify:
gh auth status # should show "Logged in to github.com"
gh auth token # should print a token starting with gho_⚠️ Important (Windows): open a NEW terminal before starting the server. If you installed or logged into
ghwhile a Command Prompt / PowerShell window was already open, that window still has the oldPATHand the GrowthKS process launched from it cannot findgh— you'll get a startup warning likegh auth token unavailable; LLM calls will failand every chat returns 401 Unauthorized. Close that terminal, open a fresh one, then start the server.
2. Install
npm install -g @chinesemachen/growthks-server-ts3. Run
Open a NEW terminal (so it picks up gh on PATH), then:
growthks # default port 5001
growthks --port 3000 # custom port
growthks --data-dir D:\my-data # custom data root
growthks --helpIf you still see a 401, inject the token explicitly so it can't depend on PATH:
:: Windows Command Prompt
for /f "tokens=*" %i in ('gh auth token') do set GITHUB_TOKEN=%i
growthks --port 3000# Windows PowerShell
$env:GITHUB_TOKEN = gh auth token
growthks --port 3000# macOS / Linux
export GITHUB_TOKEN=$(gh auth token)
growthks --port 3000A healthy startup log shows a real GitHub Models provider (not Copilot CLI (fallback)) and
no gh auth token unavailable warning.
Open http://localhost:5001 and walk through the setup panel.
4. Using it
- First-run setup — The setup panel pre-fills the KB Directory with the default. Edit it if you want a custom location. Click Done — the server runs
wiki initto scaffoldraw/andwiki/. The directory is locked once init completes. - (Optional) Add code paths — Paste an absolute path or
https://URL and click Add Path/URL. These are passed to thescan_sourceaction; no indexing happens. - Chat — Click + New Chat, ask away. Watch the planner pick an action and stream the answer.
- Add more material — Drop
.mdfiles into theraw/folder, click ⚙ Configuration in the sidebar, then click Ingest to refresh the wiki. - Edit repos later — In the Configuration panel, double-click a repo name or path to edit; changes save immediately.
Quick start (developers)
# Repo layout: src/growthks-server-ts (this) + src/growthks-web (frontend, sibling dir)
cd src/growthks-server-ts
npm install
cd ../growthks-web && npm install && cd -
# Hot-reload backend (tsx watch)
npm run dev # default port 5001
# Frontend dev server (separate terminal — proxies /api to :5001)
cd ../growthks-web
BACKEND_PORT=5001 npm run dev # Vite on :5173Build
The server's build script builds both sides — wwwroot/ (web bundle) and dist/ (compiled server):
cd src/growthks-server-ts
npm run build # builds web + server in one go
npm run build:web # only refresh wwwroot/
npm run build:server # only refresh dist/
npm start # node dist/index.js
npm run build:webshells out tonpm --prefix ../growthks-web run build:ts, which setsBUILD_TARGET=tsso Vite emits into../growthks-server-ts/wwwroot/.
Project scripts
| Script | What it does |
|--------|--------------|
| npm run build | Web bundle + server compile |
| npm run build:web | Web bundle only → wwwroot/ |
| npm run build:server | TypeScript compile only → dist/ |
| npm start | Run compiled server (node dist/index.js) |
| npm run dev | tsx watch on src/index.ts |
| npm run dev:debug | Same with Node inspector + verbose Copilot output |
| npm run prepublishOnly | Same as build — fires automatically on npm publish |
Architecture
src/
├── index.ts # Express bootstrap, graceful shutdown, Main Agent seed
├── cli.ts # `growthks` CLI entry (parses --port/--data-dir)
├── check-deps.ts # Probes for the `copilot` binary on startup
├── config.ts # Paths, env knobs, resolveAgentWorkspace()
├── db.ts # sql.js-backed DB wrapper + schema + migrations
├── errors.ts # AppError + enums (AgentStatus / RepoStatus / KbStatus / ...)
├── logger.ts # pino + pino-http
├── repository.ts # Barrel re-export of per-domain repos
├── types.ts # Shared TS interfaces
├── middleware/
│ ├── async-handler.ts # Forwards async rejections to errorHandler
│ ├── error-handler.ts # Global { error: { code, message } } responder
│ └── validate.ts # zod request validator
├── repositories/ # Per-table DB access (agent, agent-repo, chat, message, system)
├── schemas/ # zod request schemas (agent, chat, system)
├── routes/
│ ├── system.ts # KB config + init + ingest + SSE event stream
│ ├── agents.ts # Agent + repo CRUD
│ ├── chats.ts # Chat CRUD + SSE message streaming
│ └── status.ts # LLM provider / model info
└── services/ # Grouped by concern; each folder re-exported via index.ts
├── pipeline/
│ ├── chat-service.ts # Orchestrates plan → action → stream → persist
│ ├── chat-handlers.ts # Action handlers (handleScanSource / handleSearchWiki / ...)
│ ├── planner.ts # JSON-mode planner — picks scan_source / search_wiki / direct_reply / clarify
│ ├── input-triage.ts # First-pass classifier (greeting / clarify / question)
│ └── save-intent.ts # Detects "save to file" requests + target filename/format
├── retrieval/
│ ├── retriever.ts # Unified retrieval (lexical + vector) + relevance gating
│ ├── wiki-engine.ts # MiniSearch index (CJK-aware) with per-KB caching
│ ├── vector-store.ts # Embedding vector store + similarity search
│ └── embedding.ts # Embedding adapter + vector math (normalize / cosineSim)
├── answer/
│ ├── answerer.ts # Streaming wiki-grounded answers + LLM fallback
│ ├── judge.ts # Judges answer completeness → triggers scan_source follow-up
│ ├── synthesize.ts # Merges wiki + scan_source into the final answer
│ └── merge.ts # Result-merging helpers
├── icm/
│ └── icm-router.ts # IcM DRI knowledge-graph router (pure-Node port of query_example.py)
├── kb/
│ ├── system-init.ts # KB init / ingest lifecycle + main-agent path helpers
│ ├── wiki-init.ts # `wiki init` — scaffolds wiki/ tree + registers KB
│ └── feedback-curation.ts # 👍-like → distill + curate answer into the wiki (dedup)
├── realtime/
│ ├── event-bus.ts # In-process pub/sub + SSE subscriber helper
│ └── sse.ts # SSE write utilities + tool-log filter
├── llm/ # Provider clients + factory
│ ├── factory.ts # Provider resolution (Azure > OpenAI > GitHub Models > Copilot CLI)
│ ├── base-client.ts # Shared OpenAI-compatible chat/embed client
│ ├── azure-openai-client.ts / openai-client.ts / github-models-client.ts
│ ├── copilot-cli.ts # Copilot CLI `-p` + tools runner (scan_source / KB ingest)
│ ├── copilot-acp.ts # Resident `copilot --acp` process (reasoning-fallback transport)
│ ├── copilot-llm-client.ts # LlmClient over ACP (no-credential reasoning fallback; no embeddings)
│ ├── model-selection.ts # Model list + active-model selection
│ └── types.ts # Shared LLM interfaces
└── file-utils.ts # File save helpers (path-traversal safe)Deployment layout
<install root>/
├── app/ # Replaceable on update
│ ├── dist/
│ ├── wwwroot/ # Frontend bundle
│ ├── Skills/ # Copilot CLI skill definitions
│ ├── node_modules/
│ └── package.json
└── data/ # User data (persists across updates)
├── db/ # SQLite (sql.js)
├── agents/main/ # Default KB workspace (raw/ + wiki/)
└── files/ # Exported report filesRequest surface
| Route | Purpose |
|-------|---------|
| GET /api/system/config | Current KB config (mode / directory / status / paths) |
| POST /api/system/config | Set KB Directory (locked once initialized) |
| POST /api/system/init | One-shot wiki init for the chosen directory |
| POST /api/system/ingest | Re-run wiki ingest over raw/ |
| GET /api/system/events | SSE: system:status, system:log, agent:status |
| GET/POST/PATCH/DELETE /api/agents... | Agent + repo CRUD |
| GET /api/chats, POST /api/chats | List / create chats |
| GET /api/chats/:id/messages | Chat history |
| POST /api/chats/:id/messages | Send message — SSE stream (planning, tokens, done) |
| POST /api/chats/:id/messages/fallback | Direct LLM, no planning |
| POST /api/chats/:id/messages/:msgId/like | 👍 like — curate the answer into the wiki |
| DELETE /api/chats/:id/messages/:msgId/like | Remove the like + its curated wiki entry |
| DELETE /api/chats/:id | Delete chat |
| GET /api/status | LLM provider, current model, options + selection |
| POST /api/status/model | Switch the active model (must be in options) |
Configuration
Environment variables
| Variable | Default | Description |
|----------|---------|-------------|
| PORT | 5001 | Server port |
| DATA_DIR | <user home>/.growthks/data | Root for db/, agents/, files/. Also the default KB Directory |
| PUBLIC_URL | http://localhost:${PORT} | External URL used in generated file links (set behind a reverse proxy) |
| CORS_ORIGIN | http://localhost:${PORT} | Allowed CORS origin. * only for anonymous clients |
| OPENAI_API_KEY | — | OpenAI key |
| OPENAI_MODEL | gpt-4o | Default OpenAI model when no selection is saved |
| OPENAI_MODELS | — | Comma-separated list of selectable OpenAI models. Falls back to model-options.json if unset |
| AZURE_OPENAI_API_KEY | — | Azure OpenAI key. With AZURE_OPENAI_ENDPOINT, Azure takes precedence over OpenAI/GitHub Models |
| AZURE_OPENAI_ENDPOINT | — | Azure resource endpoint, e.g. https://<resource>.openai.azure.com |
| AZURE_OPENAI_API_VERSION | 2024-10-21 | Azure REST API version |
| AZURE_OPENAI_CHAT_DEPLOYMENT | — | Azure deployment name for chat (used as the model) |
| AZURE_OPENAI_EMBEDDING_DEPLOYMENT | — | Azure deployment name for embeddings. Embeddings are disabled when unset |
| GITHUB_TOKEN / GH_TOKEN | — | GitHub token for GitHub Models API. If unset, falls back to gh auth token |
| GITHUB_MODEL | gpt-4o | Default GitHub Models model when no selection is saved |
| GITHUB_MODELS | — | Comma-separated list of selectable GitHub Models. Falls back to model-options.json |
| COPILOT_CLI_PATH | copilot | Path to the standalone copilot binary (used by scan_source / KB ingest and the ACP reasoning fallback) |
| COPILOT_MODEL | — | Default Copilot CLI model. Pass-through to copilot --model. Empty / auto lets the CLI pick |
| COPILOT_MODELS | — | Comma-separated list of selectable Copilot models. Falls back to model-options.json |
| MAX_TOKENS | 4000 | Max tokens for non-streaming completions |
| EMBEDDINGS_ENABLED | true | false disables vector retrieval (lexical/minisearch only) |
| EMBEDDING_MODEL | provider default | Override embedding model (OpenAI text-embedding-3-small; GitHub openai/text-embedding-3-small; Azure uses its embedding deployment) |
| EMBEDDING_DIM | 768 | Embedding output dimension (Matryoshka shrink for text-embedding-3-*) |
| EMBEDDING_BATCH | 64 | Texts per embedding API request |
| WIKI_CASCADE | on | on = enable route_icm + search_wiki → judge → scan_source cascade; off = legacy single-action behavior |
| ICM_BUNDLE_PATH | <KB Directory>/main/icm_kg/icm_kg.bundle.json | Path to the IcM DRI knowledge-graph bundle used by route_icm. Missing file → route_icm degrades silently |
| LOG_LEVEL | info | pino level (trace/debug/info/warn/error/fatal) |
| DEBUG | — | Set 1 for verbose Copilot CLI stdout |
LLM provider priority: Azure OpenAI → OpenAI → GitHub Models → Copilot CLI (ACP). With nothing configured the server defaults to GitHub Models using the gh auth token from your gh auth login session — so reasoning and embeddings work zero-config. Chat and embeddings always use the same provider. When no API credential is available at all (no Azure / OpenAI / GitHub token and no gh auth token), the server falls back to Copilot CLI over ACP — a resident copilot --acp process driven through the Copilot CLI's own login — as the reasoning provider. This fallback has no embeddings API, so vector search degrades to lexical (MiniSearch) and cross-language / multilingual Q&A is unavailable in that mode. (Separately, the copilot CLI is also run in -p + tools mode for the scan_source agent and KB ingest; that path is independent of — and never shares a process with — the ACP reasoning fallback.)
Model picker
The left-sidebar status badge becomes a dropdown when there's a non-empty list of selectable models for the active provider. Picking one persists to SQLite (system_config.selected_model) and is used for every subsequent request — across restarts. Resolution order:
- User selection from DB (if it's still in the option list, or the list is empty)
- Provider's single-model env var (
OPENAI_MODEL/AZURE_OPENAI_CHAT_DEPLOYMENT/GITHUB_MODEL/COPILOT_MODEL) - First entry of the option list
- Hard-coded default (
gpt-4ofor OpenAI/Azure/GitHub Models,autofor Copilot CLI)
Option lists themselves come from (in priority order):
- Provider's plural env var (
OPENAI_MODELS=gpt-4o,gpt-4o-mini) model-options.jsonat the package root (openai/azure/github/copilotkeys)- Empty list (dropdown hidden, badge falls back to plain text)
A .env.example is checked in — copy to .env and edit.
Knowledge Base Directory
- Default mode —
<DATA_DIR>/agents/main/(raw + wiki live inside the standard data tree) - Custom mode —
<your path>/main/(anywhere you want, e.g. a network share or a project folder) - The directory choice is locked after the first
wiki init— change it by deletingsystem_configfrom the SQLite DB if you really need to start over.
Answer cascade (route_icm + search_wiki → judge → scan_source)
When the planner picks search_wiki, the system doesn't stop at the wiki answer. It first queries the IcM DRI knowledge graph (route_icm). If route_icm confidently routes the question (real DRI/Process hits, not the fallback path), that routing answer is authoritative and is returned directly — search_wiki, judge, and scan_source are skipped. Otherwise it falls back to querying the wiki, combining both into one knowledge answer, then running a judge → scan_source → synthesize cascade so partial coverage automatically falls back to the original raw source materials (and any attached code repos):
plan → route_icm → icmAnswer (DRI/process routing, evidence-cited)
↓
ICM confident hit? (real DRI/Process results, not the fallback path)
├─ yes → save icmAnswer, done ← skips wiki / judge / scan_source entirely
└─ no → continue ↓
+ search_wiki → wikiAnswer (streamed to user)
↓ knowledgeAnswer = icmAnswer + wikiAnswer
judge(question, knowledgeAnswer) → { status: full | partial | none, missing }
├─ full → save knowledgeAnswer, done
└─ partial / none
↓
scan_source(scan_prompt = missing) → codeAnswer (streamed)
↓
├─ codeAnswer empty → save knowledgeAnswer + "未找到更多代码层面的信息"
└─ codeAnswer non-empty
↓
synthesize(question, knowledgeAnswer, codeAnswer) → finalAnswer
→ save finalAnswer with source = "mixed"IcM DRI knowledge graph (route_icm)
route_icm reads a portable, language-agnostic data asset — icm_kg.bundle.json, built from the IcM DIR PlayBooks/ and Process/ Markdown — and derives which DRI/team owns an issue and which process to follow, each with traceable evidence (source file + line + quote). The inference is a pure-Node port of the reference query_example.py; no Python is needed at runtime. The structured routing candidates are then passed through the LLM (same as search_wiki's answerer) for precise matching and natural-language organization before streaming; if the LLM yields nothing, the raw structured routing facts are emitted as a fallback.
- Bundle location:
<KB Directory>/main/icm_kg/icm_kg.bundle.jsonby default, so it can be updated independently of the app. Override withICM_BUNDLE_PATH. - ICM-only short-circuit: a confident routing result (non-fallback DRI/Process hits) is returned on its own — the wiki search, judge, and scan_source steps are skipped, so a routing question never wrongly triggers a source scan.
- Graceful degradation: when the bundle is missing/unreadable,
route_icmcontributes nothing and the wiki/code answer stands on its own. - Rebuilding the bundle (only when the source Markdown changes):
python BCI_KnowledgeGraph/icm_kg/build_kg.py. - Validating the port:
npm run validate:icmruns the bundle's built-in cases through the TypeScript router (expects75/75).
Judge
A small LLM call evaluates the knowledge answer against the question and emits { status, covered, missing, reason }. Two cheap deterministic shortcuts skip the LLM call:
- 0 knowledge hits (no ICM results AND 0 wiki hits) →
status: none - answer contains
未找到 / 无法回答 / 信息不足 / 未在知识库中找到 / ...→status: partial
Error policy: JSON parse failure → fall back to partial (favor recall); LLM call error → fall back to full (don't compound failures).
Synthesizer
When the cascade fires AND scan_source returned content, a final LLM call merges the knowledge answer (ICM + wiki) with the source-scan answer into one coherent reply. Trusts the source answer on conflicts; preserves wikilinks and inline code blocks verbatim. On failure, falls back to a plain concatenation so the user never loses content.
Disabling the cascade
WIKI_CASCADE=off growthksThis reverts search_wiki to its legacy single-action behavior (and skips route_icm).
Triggering conditions
- Only when the planner picks
search_wiki - Skipped when
WIKI_CASCADE=off save_to_file=true— the cascade runs and produces exactly one file at the end:- Both wiki AND code answers present → file-based merge: the two
intermediate answers are written to temp files under
<files>/<agent>/.cascade-tmp/, then Copilot CLI is invoked with tool access to read both files and write the final output file in the requested format. This guarantees substantive content from BOTH sides survives and that format details (CSV BOM, JSON validity, etc.) are handled by the CLI rather than ad-hoc serialization. Temp files are deleted on success. - Only wiki content (status=full, or scan_source returned nothing) →
in-memory
synthesize()is used to produce / convert the requested format; the output is written directly with our own file-write helper. - File URL is appended to the streamed reply as
📁 完整报告已保存:<url>.
- Both wiki AND code answers present → file-based merge: the two
intermediate answers are written to temp files under
- File extension determines output format:
.md/.markdown→ Markdown;.csv/.tsv/.json/.yaml/.xml/.html→ that format; other → the extension is passed verbatim into the prompt.
Cost / latency
The cascade adds 1 LLM call (judge) in the common case; the partial/none branch adds a full scan_source + 1 LLM synthesis call. Expect:
| Case | Extra latency | Extra LLM calls | |---|---|---| | Wiki sufficient (status=full) | +1–2s | +1 (judge) | | Wiki partial → scan_source finds nothing | +30–60s | +1 (judge), 0 (synth skipped) | | Wiki partial → scan_source helps → synth | +30–90s | +2 (judge + synth) |
The judge LLM call uses the active provider's model (same as the main conversation).
Releasing (maintainers)
The package is published to npm as @chinesemachen/growthks-server-ts.
One-time setup
Log in at https://www.npmjs.com → Settings → Two-Factor Authentication (enable 2FA)
Settings → Access Tokens → Generate New Token → Granular Access Token
- Expiration: pick a reasonable window
- Packages and scopes → Permissions: Read and write
- Packages: All packages
- 2FA: check Bypass 2FA for automation (required, otherwise publish is rejected)
Copy the token, then configure npm locally:
npm config set //registry.npmjs.org/:_authToken=<token>⚠️ Never commit the token.
Cutting a release
cd src/growthks-server-ts
# Bump the version (npm publish refuses to overwrite an existing version)
npm version patch # 1.0.7 → 1.0.8 bug fix
npm version minor # 1.0.7 → 1.1.0 new feature, backwards-compatible
npm version major # 1.0.7 → 2.0.0 breaking change
# Or pin a specific version
npm version 1.2.3
# Skip git commit + tag
npm version patch --no-git-tag-version
# Publish — prepublishOnly auto-runs `npm run build` (web + server)
npm publishnpm version by default: ① updates package.json, ② creates a vX.Y.Z git commit + tag (when inside a git repo).
Forking to a different owner
| File | Where | Change to |
|------|-------|-----------|
| package.json | "name" | @chinesemachen/growthks-server-ts → @<new-owner>/<pkg-name> |
| package.json | "repository.url" | New git URL |
| .github/workflows/publish.yml | — | No change needed; uses NPM_TOKEN repo secret |
npm scopes must be lowercase and match the GitHub owner name (case-insensitive but stored lowercase).
Troubleshooting
| Problem | Fix |
|---------|-----|
| copilot: command not found | Install the standalone binary from https://aka.ms/copilot-cli, or set COPILOT_CLI_PATH |
| gh auth status shows logged out | Run gh auth login — the gh token powers the GitHub Models provider (the Copilot CLI fallback uses its own copilot login instead) |
| Setup panel keeps showing after Done | The init failed — check the System Log inside the panel for stderr from copilot |
| Wiki search returns 0 results | The wiki/ folder is empty until you drop files into raw/ and click Ingest |
| Port in use | Use growthks --port 8080 or kill whatever owns the port |
| Chinese characters mojibake in Windows console | chcp 65001 before launching |
| Permission denied on data dir | Move it: growthks --data-dir <writable path> |
| npm publish rejected with "You cannot publish over the previously published versions" | You forgot npm version patch before publishing |
