ctxdb-sync

ctxdb-sync is a local daemon for syncing DingTalk MCP, a user's Feishu documents through managed lark-cli, and local folder document trees into target ctxdb knowledge bases.

Install

npm install -g @aliyunrds/ctxdb-sync
ctxdb-sync --version

Commands

ctxdb-sync --version
ctxdb-sync start
ctxdb-sync start --reuse-running
ctxdb-sync restart
ctxdb-sync status --json
ctxdb-sync logs --kind all --tail 100 --json
ctxdb-sync diagnostics export --json
ctxdb-sync open
ctxdb-sync open --browser
ctxdb-sync run --once --dry-run
ctxdb-sync run --once
ctxdb-sync stop
ctxdb-sync service install --json
ctxdb-sync service status --json
ctxdb-sync service restart --json
ctxdb-sync service uninstall --json

start is idempotent only when the authenticated running daemon has the same version as the installed CLI. After an upgrade, a different running version returns DAEMON_VERSION_MISMATCH with both versions and leaves that process untouched; run ctxdb-sync restart, or ctxdb-sync service restart for a systemd-managed installation, to activate the installed version. start --reuse-running deliberately keeps a verified older daemon and returns success with a warning; it does not restart or replace the process. status reports installedVersion, runningVersion, updateRequired, and runtimeVerified under daemon.

Before stop, restart, or detached-to-systemd migration signals a recorded PID, ctxdb-sync verifies the process through its token-authenticated localhost runtime endpoint. A live PID that cannot prove its daemon identity returns DAEMON_IDENTITY_UNVERIFIED and is never signaled by the CLI.

run --once writes live progress and a heartbeat to stderr while it is discovering, extracting, and uploading content. The final result remains a single JSON document on stdout, so callers should judge success from the exit code and final result instead of treating any stderr output as a failure. A heartbeat is emitted every 15 seconds when the counters do not change. Add --quiet when a caller needs no progress output.

Configuration and runtime state are stored under ~/.ctxdb/sync/. Sync and daemon logs are written under ~/.ctxdb/logs/ and can be read through ctxdb-sync logs; Agent workflows do not need direct filesystem access.

Binary attachment staging uses a private per-source run directory under ~/.ctxdb/sync/tmp/runs/. Normal completion removes the directory; a later run safely reclaims directories whose owner process has exited. Legacy flat staging files from older versions are reclaimed after seven days.

For local development, start or restart the daemon from this package directory:

npm run daemon:local

ctxdb-sync open --browser opens the module-based local workbench in the default browser. The CLI passes authentication to the browser without printing the daemon token. The workbench keeps the token in sessionStorage, removes it from the visible URL, and loads only fixed local assets from src/web/; it has no CDN or frontend runtime dependency.

导出支持诊断信息

客户遇到“正在准备”、同步卡住或结果异常时，可直接点击工作台左下角的 导出诊断信息。该入口在总览、任务和历史页始终可用，包括目录仍在加载、尚无同步历史的场景。浏览器会下载一个 ctxdb-sync-diagnostics-YYYYMMDD-HHMMSS-<bundle>.json 文件；将这个文件作为附件提供给支持人员即可，不需要手工复制本机配置或日志。导出成功提示会在约 6 秒后自动收起。

无浏览器的机器可运行：

ctxdb-sync diagnostics export --json
ctxdb-sync diagnostics export --output ./customer-diagnostics.json --json

CLI 只在已运行 daemon 通过 localhost token 校验且版本与当前 CLI 一致时读取活态队列和目录加载进度；否则自动生成 persisted-only 快照，不会启动 daemon，也不会连接飞书、钉钉或 ContextDB。stdout 只返回文件路径、bundle ID、字节数、采集模式以及 partial/truncated 标记，不输出诊断正文。

诊断文件采用版本化 JSON schema，并使用两空格缩进和自然换行，便于直接阅读。单文件最多 10 MiB，daemon 与同步日志各最多保留最近 1,000 条；truncated=true 表示较早的日志或运行记录因这些容量限制没有导出，并不表示 JSON 损坏，具体保留和省略数量可查看 truncation.sections。文件可能包含任务、目录或文档名称、运行时间、结果计数、脱敏错误码等业务元数据。它不包含 API Key、OAuth/daemon token、Cookie、MCP URL、ContextDB 地址、原始 Provider/文档 ID、本机 HOME 绝对路径、文档正文或附件内容。个别状态无法读取时仍会导出其余部分，并用 section state、collectionErrors 和 CLI/UI 的 partial 提示标明。

Web first-run onboarding

When the Web UI has no config yet, it opens a four-step full-page guide instead of an unusable workbench: connect ContextDB, choose one source platform, connect that platform, then configure and validate the first sync task. The platform pages include the concrete preparation steps for DingTalk MCP, Feishu user OAuth with at most two browser confirmations, or a same-machine local folder. Feishu users do not create an app, add a bot, copy credentials, or paste document IDs for a new sync task.

The Web UI uses three distinct user-facing concepts:

• 来源平台 is the external system: DingTalk, Feishu, or a local folder.
• 同步任务 is one persistent mapping from a platform's content range to a target ContextDB knowledge base.
• 同步运行 is one execution of one or more sync tasks.

The persisted configuration and compatibility contracts still use sources, source IDs, /api/* source fields, and the ctxdb-sync source ... CLI commands. In the Web UI and its user-facing errors, those configured source objects are presented as 同步任务; this is a terminology change, not a storage or API migration.

ContextDB and sync-task validation are read-only: they do not create a knowledge base, upload documents, write checkpoints, or require a dry-run preview. Successful validation is the completion gate. Directory loading and range selection happen after entering the workbench; on a single-task page, 确认范围并同步 confirms that ongoing boundary and immediately starts the task's first real synchronization.

Automatic monitoring is off by default in the first-run guide. It may be enabled before the first sync, but scheduled sync still excludes a task until its directory range has been loaded and confirmed. The Settings home permanently exposes 配置引导 / 查看接入步骤或重新检查连接; reopening it uses the same pages, can be closed at any time, and never resets an already completed setup. Revisit changes are staged until the final action: an unchanged task is not written, while editing its onboarding fields creates a separately confirmed new task and preserves every existing task. ContextDB, the selected platform connection, and that platform's monitoring policy are saved together with the explicit task action. Existing config files without an onboarding marker continue directly to the workbench for backward compatibility.

飞书同步任务范围

Web 中的飞书同步任务只有两个互斥选项：默认的 同步我的工作区 和 导入指定目录。这也是首次引导、后续新建任务和编辑任务共用的范围选择方式。

• 同步我的工作区：默认勾选“云盘”和“知识库”，两者至少保留一个。“云盘”不再细分；进入此选项即自动加载当前账号可访问的“我的文档库”和团队知识库名称，支持搜索、多选、全选和清空。加载知识库名称不会导出正文，也不会访问云盘目录、文档内容或 ContextDB。
• 选中的知识库 ID 是当前列表的本次加载快照。“全选”只固定当次成功加载的知识库；未来新获得权限的知识库不会自动加入。列表加载失败时显示“刷新”供用户重试，绝不静默改为全量访问；知识库仍启用时不能保存失败状态。
• 导入指定目录：只输入一个飞书目录或文档 URL；导入指定目录只加载该链接定位的节点及其后代，不扫描整个工作区。它适合其他部门分享、但不在自己的知识库根列表中的内容；链接不扩大也不绕过账号已有权限。

保存任务只保存范围快照或链接，不会加载目录或启动同步。保存后仍需显式加载任务目录、选择具体节点并确认范围；只有“确认范围并同步”或后续显式同步才会开始同步。

飞书文档内附件

飞书 Provider 设置中的 同步文档内附件 默认关闭，并统一作用于全部飞书同步任务。开启后，ctxdb-sync 仍把在线文档导出为主 .docx，另外只扫描源类型为 docx 的正文块，并把 File Block / InlineFile 文件附件作为独立原文件、把 Sheet Block 和 Bitable Block 引用的整个工作簿作为独立 .xlsx 上传。Sheet 按 spreadsheet token 去重，多维表格按 app token 去重，因此同一多维表格里嵌入多个 table 也只上传一次。图片继续由主 DOCX 承载；其他嵌套在线文档、cite、whiteboard 等组件不会递归下载，旧源类型 doc 也不扫描正文附件。

每次新登录和重新授权都会申请 docx:document:readonly 与 docs:document.media:download。这两个权限只控制文档内附件能力：仅缺少它们时，飞书连接与主 DOCX 同步仍可用；开启开关后会显示重新授权提示，并把附件扫描记为可重试的部分结果。基础读取权限仍是连接就绪的必需条件。

附件扫描以父文档 revision 为边界。一个 revision 的主 DOCX、所有文件附件、整个 Sheet 工作簿和整个多维表格都稳定后，后续未变化运行不会再次读取 blocks、下载附件或导出工作簿。父 revision 变化时会重新扫描；文件附件按稳定文件身份和真实字节走普通文件的 unchanged/replacement/直接或分片上传链路，Sheet 与多维表格则按父 revision 重新导出并走安全替换，即使 .xlsx 字节恰好相同。V1 不独立轮询附件、Sheet 或多维表格的变化，也不因块被移除或 token 被替换而删除旧 ContextDB 子文档；关闭开关同样不会清理已上传子文档。

Common local configuration:

ctxdb-sync config set ctxdb.apiKey '<ctxdb-api-key>'
ctxdb-sync config set ctxdb.userId '<ctxdb-memory-user-id>'
ctxdb-sync config set ctxdb.targetKbName '<target-kb-name>'
ctxdb-sync config set dingtalk.documentEndpoint 'https://mcp-gw.dingtalk.com/server/<server-id>?key=<api-key>'
ctxdb-sync config set dingtalk.sheetEndpoint 'https://mcp-gw.dingtalk.com/server/<sheet-server-id>?key=<api-key>'
ctxdb-sync config set dingtalk.workspaceId '<space-url-1>,<space-url-2>'
ctxdb-sync schedule set dingtalk --every-minutes 30 --json
ctxdb-sync config set sync.maxConcurrency 10
ctxdb-sync config set sync.maxAcquisitionConcurrency 5
ctxdb-sync config set sync.verifyUnchangedTargets true
ctxdb-sync config show

Each provider owns one simple automatic-sync policy. Choose either a continuous whole-minute interval or one to six daily times in the daemon machine's local timezone:

ctxdb-sync schedule set feishu --every-minutes 31 --json
ctxdb-sync schedule set feishu --daily-at 02:00 --daily-at 09:00 --daily-at 18:00 --json
ctxdb-sync schedule show feishu --json
ctxdb-sync schedule disable feishu --json

Intervals keep their own continuous cadence, so 31 minutes means every 31 minutes from the saved boundary rather than minute 0/31 of each hour. Daily values must use HH:mm; duplicates are removed, values are sorted, and more than six distinct times are rejected. schedule set updates a running local daemon when reachable but never starts a sync by itself. Cron, weekdays/workdays, holidays, time windows, explicit timezone selection, stacked rules, and per-task schedules are intentionally not exposed. Use interval mode for denser regular schedules.

Agent-only CLI workflow

An autonomous agent can configure and verify synchronization without reading or editing the config file and without using the browser UI. The source boundary is a complete configured root: DingTalk uses the configured node, a new Feishu source uses a pasted URL and loads that linked subtree for range selection, and a local folder source includes one complete directory. Loading every accessible Feishu root is available only through the explicit all-accessible choice below. These CLI compatibility choices are not a third Web option: Web uses the two choices in 飞书同步任务范围. Use the Web UI when a human needs to select arbitrary descendants and confirm the chosen range.

Discover the versioned CLI contract first:

ctxdb-sync capabilities --json

Configure a DingTalk source idempotently:

export CTXDB_API_KEY='<ctxdb-api-key>'
export DINGTALK_DOCUMENT_MCP='<dingtalk-document-mcp-url>'

ctxdb-sync config set ctxdb.apiKey --env CTXDB_API_KEY --json
ctxdb-sync config set ctxdb.targetKbName '<target-kb-name>' --json
ctxdb-sync config set dingtalk.documentEndpoint --env DINGTALK_DOCUMENT_MCP --json
ctxdb-sync source apply engineering-docs \
  --type dingtalk-mcp \
  --workspace '<space-url-1>,<space-url-2>,<node-url>' \
  --target-kb '<target-kb-name>' \
  --json
ctxdb-sync config validate --source engineering-docs --json
ctxdb-sync source test engineering-docs --json
ctxdb-sync run --once --source engineering-docs --dry-run --json

Connect a Feishu user and create a URL-scoped source idempotently (recommended):

No enterprise app, App ID, App Secret, bot, or manually copied permission JSON is required. The managed PersonalAgent flow asks the user to approve the product-defined read scopes in the browser. source test verifies the active user plus Wiki/Drive reachability but does not export online documents; the targeted dry-run below performs the first export-permission check without uploading to ContextDB.

ctxdb-sync feishu connect --json
ctxdb-sync feishu status --json
ctxdb-sync source apply product-feishu \
  --type feishu \
  --scope-mode links \
  --scope-links 'https://example.feishu.cn/wiki/<node-token>' \
  --target-kb '<target-kb-name>' \
  --json
ctxdb-sync config validate --source product-feishu --json
ctxdb-sync source test product-feishu --json
ctxdb-sync run --once --source product-feishu --dry-run --json

Use all-accessible only when loading every directory visible to the connected account is an explicit choice:

ctxdb-sync source apply product-feishu-all \
  --type feishu \
  --scope-mode all-accessible \
  --target-kb '<target-kb-name>' \
  --json

There are two separate boundaries. lark-cli OAuth determines what the connected account is allowed to access; URL scope determines which shared directory or document ctxdb-sync actually loads so the user can choose a range. A URL can locate another department's shared directory even if it is absent from root listings, provided the connected account already has permission. It neither narrows the OAuth grant itself nor bypasses that permission. Loading a directory reads metadata only; extraction and ContextDB upload begin only after the user selects a workbench range and confirms it.

feishu connect installs the pinned private lark-cli runtime when needed, creates or adopts a dedicated PersonalAgent profile, prints the browser confirmations to stderr, and persists only verified profile metadata. Use --profile <name> to adopt a discovered profile, --create to force a new dedicated profile, and feishu reconnect if authorization expires. For a legacy App-based config, connect returns migrationRequired plus the real directory difference without changing the active config; review it, then run feishu migrate --profile <name> --confirm --json (or use the Web confirmation button) to repeat discovery, run an isolated zero-write continuity preview, and atomically commit v2. source apply returns created, updated, or unchanged, so the complete Agent workflow is safe to repeat. config validate is local and read-only; source test verifies the active user identity plus bounded document/ContextDB access; targeted --dry-run uploads nothing. Advanced Wiki IDs and Drive folder tokens remain supported only for migrated explicit-root sources.

Configure a local folder idempotently:

ctxdb-sync source apply local-docs \
  --type local-folder \
  --root-path '/absolute/path/to/folder' \
  --target-kb '<知识库名称>' \
  --json
ctxdb-sync source test local-docs --json
ctxdb-sync run --once --source local-docs --dry-run --json

The root path must be absolute and readable on the same machine as ctxdb-sync. All non-hidden regular files are discovered; hidden entries, symlinks, and common incomplete cloud placeholders are skipped. An Obsidian Vault is a local-folder compatibility scenario: .obsidian itself is skipped while the Vault's non-hidden content remains eligible, with no plugin, account, or REST API required. The workbench requires the directory range to be selected and confirmed before scheduled synchronization includes the source. Moving or renaming a file creates a new target identity, and deleting a local file never deletes a ContextDB document.

Black-screen Linux service

On a Linux server with systemd, install ctxdb-sync as a service for the current user after the CLI configuration validates:

ctxdb-sync config validate --json
ctxdb-sync service install --json
ctxdb-sync service status --json
ctxdb-sync logs --kind all --tail 100 --json

The unit is installed at ~/.config/systemd/user/ctxdb-sync.service. It runs the installed CLI's foreground daemon, uses the same ~/.ctxdb/sync/config.json, restarts on failure after five seconds, and contains no credentials. Repeating service install updates the same unit. If an older detached daemon from ctxdb-sync start is running, installation stops it before systemd starts the managed process.

Provider runs are serialized with a filesystem lock under ~/.ctxdb/sync/locks/. A scheduled daemon run and a manual run --once for the same provider therefore wait for each other when they are separate processes; different providers may still run independently. Inside one daemon, interval schedules stay on their saved fixed cadence and daily schedules follow the daemon machine's local wall clock. If the provider is still running or its scheduled preflight has not finished when an occurrence arrives, it is recorded as provider-busy and skipped. Occurrences missed while busy or while the daemon is stopped are not queued or backfilled; the daemon advances to the next future occurrence.

For the user manager to start at boot and remain alive after logout, enable linger once when service install returns LINGER_DISABLED:

loginctl enable-linger "$USER"

This can require administrator policy on some distributions; ctxdb-sync reports the warning but never invokes sudo. Remove the service without deleting sync configuration, checkpoints, or logs:

ctxdb-sync service uninstall --json

service commands intentionally return UNSUPPORTED_PLATFORM on macOS and Windows. ctxdb-sync start/stop/restart remain available for local foreground-style operation, but systemd is the supported long-running Linux lifecycle.

Operational JSON commands use one envelope:

{
  "schemaVersion": 1,
  "ok": true,
  "command": "status",
  "result": {}
}

logs accepts --kind sync|daemon|all and --tail 1..1000. Every returned line is redacted again, including daemon stderr. The sync log and detached-daemon log rotate at 5 MiB with three backups; CLI and Web diagnostics read across those backups, while service uninstall continues to preserve the bounded log set. JSON errors use the same envelope with ok: false, a stable error.code, exit 1 for operational failures, and exit 2 for command-usage errors.

Configuration boundaries:

• Fixed service endpoint: ContextDB always uses https://context-database.aliyuncs.com; it is not user-configurable.
• Default connection: ctxdb.apiKey, optional Memory destination ctxdb.userId, ctxdb.createKbIfMissing, ctxdb.requestTimeoutMs, and the default ContextDB target. Every sync.* option remains global.
• Provider-scoped: the DingTalk MCP connection, the verified Feishu user profile, providerSchedules, runtime state, and checkpoint namespaces. Local folders have no shared connection section; each source owns its rootPath. Each provider owns one schedule/runtime lifecycle.
• Source-scoped: discovery range, include/exclude filters, and the task's explicit ContextDB Workspace and knowledge-base identity. ContextDB credentials, connection settings, create policy, and timeout cannot be overridden directly per source.
• Source name and id remain unique across the complete config because they are selectors and attribution keys, not provider-local labels.

ContextDB Workspace compatibility:

• Existing legacy single-Workspace tasks without a saved selector remain compatible.
• Web settings and the CLI may add extra Workspace connections; each connection stores one API Key and one default KB.
• Every Web or CLI task save copies the selected Workspace's current default KB ID/name into the task; custom targets persist their own paired ID/name.
• Changing a Workspace default affects future task saves only. As a compatibility migration, the same update first pins legacy inherited tasks to the old KB.
• Removing a connection pauses and preserves referencing tasks; it does not delete remote knowledge bases or tasks.

ContextDB Workspace CLI

The CLI manages the same stable Workspace connections as the Web workbench. It never accepts an API key as a positional argument or literal option value: use a named environment variable or one piped stdin line instead. Results and errors never print the key.

# Read-only: refs, default status, and default KB identity only (no API keys).
ctxdb-sync workspace list --json
ctxdb-sync workspace kb list default --json

# Create a saved Workspace with an already known KB identity.
export CTXDB_TEAM_API_KEY='<team-api-key>'
ctxdb-sync workspace save \
  --api-key-env CTXDB_TEAM_API_KEY \
  --target-kb 'team-docs' \
  --target-kb-id 'kb-team-docs' \
  --json

# Update an existing Workspace target while retaining its saved API key.
ctxdb-sync workspace save <workspace-ref> \
  --target-kb 'team-docs-v2' \
  --target-kb-id 'kb-team-docs-v2' \
  --json

workspace kb list is the only read-only remote lookup. workspace save stays local when given a target ID. Add --create-target only when intentionally resolving or creating the named KB remotely; it cannot be combined with --target-kb-id:

printf '%s\n' "$CTXDB_TEAM_API_KEY" | \
  ctxdb-sync workspace save \
    --api-key-stdin \
    --target-kb 'new-team-docs' \
    --create-target \
    --json

The new Workspace ref is returned as result.workspace.ref. Switching the default never rewrites existing task bindings. Removing a Workspace pauses enabled tasks that reference it and retains their saved Workspace and KB identity; removing the current default while another connection remains requires a replacement:

ctxdb-sync workspace set-default <workspace-ref> --json
ctxdb-sync workspace remove <workspace-ref> --next-default <remaining-workspace-ref> --json

Bind a source explicitly with --ctxdb-workspace, which is intentionally distinct from DingTalk's --workspace source-range option. A custom source target always requires the name and stable ID together. To replace an existing target override with a snapshot of the Workspace's current default, use the explicit inheritance form:

ctxdb-sync source apply team-folder \
  --type local-folder \
  --root-path '/absolute/path/to/team-folder' \
  --ctxdb-workspace <workspace-ref> \
  --target-kb 'team-custom-docs' \
  --target-kb-id 'kb-team-custom-docs' \
  --json

ctxdb-sync source apply team-folder \
  --ctxdb-workspace <workspace-ref> \
  --inherit-workspace-target \
  --json

For an intentional per-source remote KB creation, use --ctxdb-workspace <workspace-ref> --target-kb <name> --create-target. Calls without any of these new binding flags preserve the existing CLI behavior: newly created sources pin the current default Workspace and its current default KB.

The browser's top-level Settings action always opens the global settings center. Global ContextDB/upload settings, each provider's shared connection and schedule, and each concrete source are presented as separate ownership levels. A global save submits only the global block; a provider save submits only that provider's shared fields, schedule, and sources. Unsaved drafts in other scopes are neither submitted nor used for validation, and the server preserves their previously saved blocks. Navigating away from dirty settings requires an explicit discard confirmation. The merged complete config still receives structural validation before it is written.

All DingTalk sources use the same MCP connection; sources only define which workspace or folder entry to scan, optional filters, and whether they override the target KB.

Code-owned DingTalk MCP contracts

DingTalk tool names are implementation contracts, not configuration. Document access always uses list_nodes, get_document_info, get_document_content, and download_file; spreadsheet access always uses the three tools below. Changing a name without changing and testing its request/response adapter is unsupported.

dingtalk.sheetEndpoint is optional and must point to the official DingTalk spreadsheet MCP endpoint, not the document MCP endpoint. The built-in adapter discovers capabilities once per endpoint and requires these exact tools:

• get_all_sheets to enumerate the complete workbook in provider order;
• get_sheet to read each worksheet's stable sheetId, visibility, used range, and bounded merged-range metadata;
• get_range_as_csv to read explicit A1 row windows using formatted values without row-number annotations.

One selected axls workbook remains one source-tree item, but a real read emits one complete CSV artifact per worksheet, including hidden worksheets. Each CSV is stored below the workbook's logical directory and is checkpointed by stable sheetId, so worksheet rename or reorder does not create a new recovery identity. An authoritatively empty worksheet creates no placeholder CSV. An incomplete listing/read is non-authoritative: prior worksheet targets are retained and missing worksheets are not inferred as deleted. Run JSON exposes separate bounded summary.dingtalkSheets.workbooks and summary.dingtalkSheets.worksheets counts; the local UI presents the same distinction without estimating percentages.

Older configs may still contain sources[*].toolNames. ctxdb-sync ignores that entire object during normalization and runtime selection, regardless of whether it contains historical defaults, custom strings, or malformed values. start, foreground daemon/service startup, and run --once do not classify, migrate, or rewrite the config because of this field, and no restart is required. A later supported config save may naturally omit the ignored unknown field; there is no dedicated cleanup step.

Automatic monitoring and runtime state are owned once per provider. DingTalk, Feishu, and local folder sources each share their provider's timer, lock, queue, and last-run record. Sources of one provider run sequentially in configuration order, while different providers may run concurrently. Use the structured schedule commands for new automation. The scalar CLI aliases dingtalk.schedule.*, feishu.schedule.*, and local.schedule.* remain compatible with interval policies; legacy schedule.* remains a compatibility alias that updates all providers. A config saved with type=daily-times must be converted back to an interval policy before running an older ctxdb-sync binary that predates daily schedules.

Feishu change detection is scheduled reconciliation, not an event subscription. ctxdb-sync intentionally registers no lark-cli webhook/event handler, and no event output may write checkpoint success. Every checkpoint transition still comes from a complete discovery/extraction/upload result. A new document under a selected folder is uploaded on the next successful run; an edit uses the existing safe replacement flow. When document-body attachment sync is enabled, its parent completion marker is written only after authoritative enumeration and stable primary/child decisions; permission, acquisition, upload, accepted-polling, cancellation, or deferral leaves the marker incomplete so the same revision retries while settled children reuse their own checkpoints. Missing, inaccessible, trashed, deselected, renamed, or removed child references never cause automatic ContextDB deletion. A connection-wide authorization or lark-cli contract failure pauses the Feishu provider schedule with one reconnect error; successful reconnect re-arms the saved policy without clearing checkpoints.

A provider run continues after a concrete source fails. Mixed results return ok: true with outcome: "partial" and retain each source's error in sourceResults plus summary.bySource; if every requested source fails, the result is ok: false with outcome: "failed". Provider selectors such as /api/sync?source=dingtalk-mcp run the whole provider, while a concrete source name remains available as a compatibility selector.

Multi-source configuration:

# Add one DingTalk source containing multiple workspaces that still writes to ctxdb.targetKbName.
ctxdb-sync source add ops-docs \
  --workspace '<space-url-1>,<space-url-2>'

# Add a DingTalk folder directly as a source entry.
ctxdb-sync source add ops-folder \
  --workspace 'https://alidocs.dingtalk.com/i/nodes/<nodeId>'

# Add a source that writes to its own KB.
ctxdb-sync source add rds-params \
  --workspace '<workspaceId>' \
  --target-kb 'rds-params-kb'

# Restrict a source to one or more DingTalk directories.
ctxdb-sync source set rds-params rootFolderIds 'https://alidocs.dingtalk.com/i/nodes/<nodeId>'

# Connect a Feishu user and add a URL-scoped source (recommended).
ctxdb-sync feishu connect --json
ctxdb-sync source apply product-wiki \
  --type feishu \
  --scope-mode links \
  --scope-links 'https://example.feishu.cn/wiki/<node-token>' \
  --target-kb 'product-kb'

# To explicitly load every directory visible to the connected account instead:
ctxdb-sync source apply product-wiki-all \
  --type feishu \
  --scope-mode all-accessible \
  --target-kb 'product-kb'

# Load metadata in the workbench, then select and confirm the actual range.

# Add one complete local folder.
ctxdb-sync source add personal-notes \
  --type local-folder \
  --root-path '/absolute/path/to/notes' \
  --target-kb 'notes-kb'

ctxdb-sync source list --json

Workspace/source entry fields accept comma- or newline-separated raw workspace IDs, alidocs.dingtalk.com/i/spaces/... links, raw node IDs, or alidocs.dingtalk.com/i/nodes/... folder links. One source may therefore scan several knowledge bases and directories together. The canonical config stores all spaces in ordered workspaceIds, keeps workspaceId as the first-item compatibility alias, and stores directories in rootFolderIds. Explicit root-folder fields accept node IDs or node links only; use the workspace/source entry for space IDs.

sync.maxConcurrency controls only the bounded ContextDB upload worker pool. It defaults to 10 and is normalized into the range 1..20.

sync.maxAcquisitionConcurrency controls remote file acquisition separately. It defaults to 5 and is normalized into the range 1..10. DingTalk documents, sheets, and ordinary files share one DingTalk limit across all DingTalk sync tasks; Feishu online exports and ordinary Drive downloads likewise share one Feishu limit across all Feishu tasks. The two Providers have independent controllers, so DingTalk saturation or cooldown does not consume Feishu slots. Directory traversal stays serial within a task and waits for an active Provider cooldown without occupying a file-acquisition slot. Local-folder tasks do not use this remote limit.

sync.verifyUnchangedTargets controls opt-in recovery for ContextDB documents deleted outside ctxdb-sync. It defaults to false. When enabled, every normal manual or scheduled sync checks each otherwise-unchanged owned document by its exact ContextDB document ID. A confirmed document not found clears only that checkpoint-owned locator and recreates the current source artifact; permission errors, timeouts, rate limits, connection failures, 5xx responses, and unrecognized errors fail closed without uploading a possible duplicate. Enabling the option therefore adds up to one ContextDB detail request per otherwise-unchanged selected document on every run. Preview and --dry-run do not perform the check or repair.

File admission is based on the lower-cased final filename extension: text/code is limited to 20 MiB, complex documents/images to 500 MiB, audio to 10 MiB, and video to 2 GiB. Existing files at or below 500 MiB keep the original direct multipart transport. Supported video above 500 MiB and at most 2 GiB uses ContextDB's chunk transport automatically; only one chunk transfer is active at a time, while ordinary direct uploads continue to use sync.maxConcurrency. Large or unknown-size video acquisition requires enough temporary disk for the expected artifact plus a 512 MiB reserve. The disk check and materialization of these large candidates are serialized in-process so concurrent providers cannot both spend the same free-space observation. Missing or unknown extensions are unsupported, including dotless Dockerfile and Makefile; dotted names such as foo.dockerfile, .gitignore, and .editorconfig follow their extension policy. sync.maxFileBytes remains readable for compatibility but no longer overrides these category limits during normal sync.

Normal-sync ordinary file downloads and ContextDB text/binary uploads use an absolute timeout budget derived from expected bytes: 30 seconds of fixed overhead plus transfer time at 128 KiB/s, clamped to a 60-second minimum and a two-hour maximum. Exact UTF-8, filesystem, or provider-declared sizes are preferred; unknown-size video may use its 2 GiB category limit as the conservative acquisition budget, while other file types keep the direct-transfer/category ceiling. A chunk ticket is treated as a single bounded session: the service ticket lasts one hour, sync uses only a 55-minute request window, sends each part and completion at most once, records the exact owned ContextDB document before the first part, and never persists the signed ticket or part ETags. The legacy API has no part-list/resume or ticket-renew endpoint, so interrupted uploads restart from byte zero only after the owned document is confirmed absent or safely deleted. On an ambiguous response or restart sync reconciles only that exact document ID. An exact owned uploading document is deleted by ID when the service permits it; until deletion is confirmed, or when the status is unknown/unavailable, the checkpoint remains chunk-cleanup-pending instead of being reported as success or adopted by name. The deployed legacy init contract also fixes the remote file_path to the root for chunk uploads; sync keeps the canonical provider path in file_name, but cannot make large-media remote directory metadata identical to the direct endpoint without a kernel API change. Generic API requests keep ctxdb.requestTimeoutMs, Feishu online-document export keeps its separate deadline, and ContextDB ingestion polling keeps sync.pollTimeoutMs.

Remote Providers finish directory discovery first. After that, selected DingTalk records are read or downloaded with bounded concurrency, while Feishu online exports and ordinary Drive downloads use the same bounded Feishu acquisition pool. Each Provider defaults to 5 active acquisitions and remains independent of the ContextDB upload pool. Feishu export downloads additionally retain a local rolling admission limit of 90 starts per 60 seconds, leaving headroom below Feishu's 100-per-minute endpoint limit.

For idempotent DingTalk/Feishu acquisition reads, ctxdb-sync makes at most 4 attempts including the initial request. Explicit Retry-After is honored up to 60 seconds; otherwise retries use jittered 1-, 2-, and 4-second baselines. An explicit Provider rate limit pauses new requests for that Provider and halves its effective acquisition limit once for the cooldown epoch (the default 5 becomes 2). Ten consecutive successful acquisitions restore one slot at a time. Authentication, permission, invalid-request, unsupported, file-size, and cancellation failures are not automatically retried, and this policy never retries a ContextDB upload POST.

Every completed remote acquisition enters the shared upload pool immediately; it does not wait for the same source to finish acquiring all remaining content. The global in-flight upload count still cannot exceed sync.maxConcurrency, independently of the acquisition limit.

If explicit Provider throttling exhausts the bounded retry budget, ctxdb-sync stops dispatching new acquisition work for that source, drains already acquired artifacts through upload/checkpoint handling, and records the remaining count once as deferred with reason provider-rate-limited. The run is partial, including when zero files completed; the Web UI shows an amber waiting or deferred state rather than one red failure per unstarted file. “重试未完成文件” starts the task's normal synchronization again, so completed checkpoint records are skipped and only unfinished work is reacquired. A terminal authentication, permission, configuration, or source-wide failure still takes precedence and remains failed.

Checkpoint idempotency stores each source node's provider update time when the remote listing provides it. A later run skips reading and hashing unchanged source content when that value is unchanged; if it is missing or changed, ctxdb-sync reads the content and uses the local content hash as the final equality check.

Within one configured DingTalk source, overlapping workspace and folder entries are grouped by the provider's remote document identity after selection/filtering and before content extraction. The UI still shows every configured entry, but one remote document is read, hashed, and uploaded at most once. Different remote documents are never merged merely because their names or bytes match.

Each source document uses this three-part checkpoint key:

<targetKbName>::<providerType>::<providerDocumentId>

The configured source ID is intentionally not part of the key. Two overlapping sources from the same provider therefore share one checkpoint for the same provider document and upload it only once. providerType keeps DingTalk, Feishu, and local folders isolated. Local file identity also includes a hash of the folder's canonical root path, so equal relative paths in different roots remain separate while the same root configured twice de-duplicates.

New or re-extracted documents use their complete logical path as the ContextDB document name:

DingTalk:     /dingtalk/<scope-id>/<folders...>/<output-filename>
Feishu Drive: /feishu/drive/<root-folder-token>/<folders...>/<output-filename>
Feishu Wiki:  /feishu/wiki/<space-id>/<folders...>/<output-filename>
Local folder: /local-folder/<folder-name>-<root-hash>/<relative-path>

These paths always use POSIX / separators. Each provider path segment is escaped reversibly, so distinct names such as A:B and A?B cannot collapse, and a literal title A/B remains distinct from file B inside folder A. DingTalk online exports keep the reserved .~export marker, for example Runbook.adoc -> Runbook.adoc.~export.md. Feishu online documents instead use the visible title plus the output extension exactly once: Runbook -> Runbook.docx, Runbook.docx -> Runbook.docx, and Sales -> Sales.xlsx. Ordinary Feishu files keep their filenames.

When two different Feishu provider document IDs require the same complete target path, ctxdb-sync keeps one clean name and gives the other a stable suffix before the extension, for example untitled (4f32c8ab).docx. The suffix is allocated only from provider-wide checkpoint ownership proven by ctxdb-sync; an unexplained ContextDB 409 remains target-conflict and is never auto-renamed. The configured source ID is not part of document identity, so the same Feishu provider document found through overlapping sources keeps one assignment and still uploads only once. Complete paths let equal filenames in different directories upload independently, while the provider prefix prevents cross-provider name collisions. All downloaded temp files, including DingTalk MCP path responses, are copied into ctxdb-sync storage under a provider plus stable raw-remote-ID hash, so external path reuse or lossy local filename sanitization cannot overwrite another document.

DingTalk's 知识库·... and 目录·... nodes are display-only containers and never enter these logical names. Adding another scope therefore cannot rename a document already discovered through a workspace. A folder-only source remains rooted at its real folder token because ctxdb-sync deliberately does not query remote ancestors.

The complete-path naming upgrade preserves checkpoint economics and does not batch-migrate history. New documents use the complete path immediately. Once a checkpoint successfully owns a ContextDB target, its first successful name and target directory normally remain sticky even when the source content changes and the target is replaced. A changed provider update time whose extracted content hash is still equal remains a skip. target-conflict does not own a target and therefore does not pin a stale name.

Feishu's binary-export and clean-name migration is the explicit exception. On the first sync after this upgrade, an existing Feishu online target such as Runbook.~export.md or Runbook.~export.docx migrates to Runbook.docx (and Sheets migrate to .xlsx) while preserving checkpoint identity and target directory. The new target is uploaded first and recorded beside the old target. A pending-review or interrupted new upload leaves the old target primary; a successful new target is promoted before old-target cleanup. Cleanup is resumable and never causes another upload. A pre-accept export/upload failure or unexplained 409 performs no delete and leaves the old primary checkpoint unchanged. Ordinary content changes with an unchanged name keep the existing replacement behavior. DingTalk naming and representation do not migrate.

A ContextDB 409 is a warning for that single document, not a source, provider, or whole-run failure. ctxdb-sync records target-conflict, continues the run, and shows the completed run with the number of documents not uploaded. It sends one POST with the canonical name and never retries automatically with a hash suffix. Ordinary runs do not POST an unresolved conflict again while both its canonical name and content hash are unchanged.

Run details expose one explicit overwrite operation in two scopes: each eligible conflict row shows 覆盖重传, and a same-source conflict group with recorded canonical names shows 批量覆盖 N 个. Both actions send the stable configured-source ID, selected remote document IDs, and exact ContextDB names through the same confirmation flow. The confirmation lists every exact name that may be deleted; the server then revalidates that each checkpoint is still target-conflict and still carries the confirmed name.

An overwrite run is selected-document-only: only the confirmed remote IDs are extracted, returned in document decisions/results, and eligible for checkpoint adoption or writes. Unselected successful, unchanged, or conflicting documents remain untouched even if their provider update time changed. For each selected item, no exact remote name match uploads directly; exactly one match gets one delete attempt and a bounded deletion wait, followed by exactly one upload attempt even if deletion fails or times out. Explicit confirmation intentionally allows that delete attempt even when the target is pending_review or graph processing. The upload result is authoritative: success writes the new target ID, 409 keeps target-conflict, and another upload error is recorded as the failure. Multiple exact matches, a missing document ID, or a changed confirmed name still leaves only that selected item conflicted and continues the batch. Ordinary synchronization is unchanged: content replacement still performs its existing detail/status safety check before deleting an owned target. The overwrite operation never touches successful/unchanged checkpoints, never runs automatically, and is not a global force-resync.

Remote deletion recovery deliberately keeps the same economic boundary. If a ContextDB target is deleted while the source remains unchanged, ctxdb-sync does not query every checkpoint target and cannot discover the deletion. After the source content changes, re-extraction enters replacement handling; a 404 or ContextDB's explicit 400 document not found response for the old target is treated as a missing target, so the current document is uploaded as new and its checkpoint is updated. The same two responses also complete an already-requested deletion wait. This recognition is limited to old-target replacement/deletion checks; unrelated 400 responses preserve their original error and never become a new upload.

ContextDB pending_review means the target upload succeeded and is waiting for a platform user to review it. Runs therefore count it with successful uploads and annotate it as 待平台审核; it is not shown as 需处理 and gets no review-specific overwrite action. If the provider document changes during review, ctxdb-sync preserves the target and an internal changed-needs-replace checkpoint, reports that the latest source update will sync after platform review, and retries the normal safe replacement flow on later runs.

If an upload was accepted but its ingestion poll failed, the next run resolves that accepted target before processing a newer provider edit, preventing a second target from being posted during the race. Other changed-needs-replace records are also retried through the same detail-check, safe-delete, and replacement flow on later runs; they never fall through to an unguarded new upload.

When DingTalk content extraction returns a transient "content is not ready, retry later" error, ctxdb-sync retries that item with short backoff before marking it failed. Other extraction or upload errors keep their normal fail-fast behavior.

While a daemon sync is running, GET /api/status includes a bounded daemon.current.queue summary. Large queue stages can be browsed on demand with:

curl -H "x-ctxdb-sync-token: <token>" \
  "http://127.0.0.1:<port>/api/queue?stage=pending&offset=0&limit=100"

Supported source adapters are DingTalk MCP, Feishu user OAuth through managed lark-cli, and read-only local folders. The persisted provider namespace remains feishu-openapi only for checkpoint and config compatibility. DingTalk OpenAPI/Drive mode and direct cloud-folder protocols remain outside the current scope.