@codebam/dsh-thinking-auditor

A claim/evidence auditor for live DeepSeek Harness reasoning traces.

It observes the model's reasoning-delta stream without changing the request, checks what the trace claims against what the session actually saw, asks the same provider/model route for a structured second opinion in a session-less call the monitored agent never sees, and lets deterministic code decide whether to force verification, gate an irreversible action, or (only if explicitly configured) cancel.

It is not a lie detector. It does not read intent and does not label a model dishonest. Chain-of-thought is an unfaithful generated artifact, not a window into hidden state, so the plugin reports only claim-to-evidence relations: supported, unsupported, contradicted, or speculative.

Why this shape

• Covert capture. A Cordis listener on agent/assistant-stream copies reasoning deltas into a bounded per-session ring window and passes every chunk through untouched. Neither the prompt nor the request route changes, so the monitored model has no new input to react to.
• Mechanical evidence first. The session log is the harness's model-visible source of truth. The plugin maintains a bounded ledger of tool/call, tool/result, and user messages from committed session/events. Claims that require a tool result but have no antecedent are settled in code, not by a model. This is the highest-precision signal the design offers.
• Same-model, isolated auditor. The auditor call uses the session's latest logged request/header route (so it follows model changes) but omits sessionId and purpose. It attaches to no session, enters no persistence log, and emits no agent/assistant-stream frame. The monitored model never sees the auditor's request or response.
• Deterministic gates. The auditor's structured JSON enriches a verdict; it cannot clear a mechanical contradiction or a missing tool-call antecedent. A small pure policy module owns escalation.
• Separate audit store. Verdicts live in a bounded in-memory store and, by default, a private 0600 JSONL file beside the harness home. They are deliberately not appended to the session log: an out-of-tree session event without the envelope's ignorable marker can make a log unreadable on resume, and Session.append() cannot set that marker.

How a verdict is produced

1. Capture. Every reasoning-delta is appended to the session's pending window. At windowChars, after a completed tool result, at an attempt end, or before a turn closes, the window is taken and cleared exactly once.
2. Deterministic pass. src/claims.mjs extracts claims and anchors (quoted excerpts, paths, URLs, versions, quantities, action verbs), then joins them against the ledger:
   • an asserted tool result with no matching tool call in the turn → unsupported / high / noMatchingToolCall;
   • a success claim contradicted by the paired result's failure markers → contradicted / high;
   • a quoted file/output excerpt absent from every evidence source → unsupported / high (or contradicted when the matching read result exists without it);
   • hedging, plans, and predictions → speculative, never a fabrication.
3. Auditor pass. The same route receives the user-request excerpt, the bounded evidence digest, the mechanical findings, and the trace window. Trace and evidence are XML-escaped and declared untrusted data; the system prompt explicitly refuses to follow instructions found inside them.
4. Gate. Mechanical findings stay authoritative. Trailing-window hysteresis, the configured maxTier, and the reversibility of the next action decide the actuator response.

Install

npm / profile dependency (0.1.1+)

Version 0.1.1 declares dsh.bundle.patch, so installing it through the dsh plugin command auto-activates it:

dsh plugin --profile <profile> add @codebam/dsh-thinking-auditor

dsh plugin appends the package to the profile's dsh.profile.bundles, and the bundle patch mounts the auditor row with the defaults (enabled: true, gates.maxTier: verify, store.persist: true). Restart dsh if the running profile does not hot-reload its bundle list.

To change defaults per profile without editing the package, either tune the thinking-audit settings namespace or address the row by id in ~/.dsh/profiles/<profile>/cordis.patch.yml (later layers win):

- id: thinking-auditor
  config:
    gates:
      maxTier: block # observe | verify | block | cancel

0.1.0 has no bundle metadata, so it requires the explicit row below or an upgrade.

Source / pinned-path install

When loading the plugin from a checkout or an absolute path instead of a profile dependency, add the row yourself (~/.dsh/profiles/<profile>/cordis.patch.yml):

- insert:
    - id: thinking-auditor
      name: '@codebam/dsh-thinking-auditor' # or an absolute path to index.mjs
      config:
        enabled: true
        gates:
          maxTier: verify # observe | verify | block | cancel
        store:
          persist: true

The stock dsh base bundle already mounts the llm, sessions, and tools services this plugin injects. Node resolves the plugin's @deepseek-ai/* peers from the profile's node_modules; when loading by absolute file path, make that resolution available next to index.mjs (the NixOS route below symlinks it).

A live settings section with the namespace thinking-audit overlays the row when dsh-settings-file is mounted; use it for thresholds and gates you want to change without restarting dsh. Unknown or invalid values fail the write, never silently drift.

Configuration

Top level:

| Field | Default | Meaning | |---|---|---| | enabled | true | Master switch; false makes apply() a no-op. | | auditOnToolResult | true | Audit the pending window after a completed tool result. | | auditOnTurnStop | true | Audit/await a verdict before a turn closes (T1/T3 decision point). | | auditOnAttemptEnd | true | Audit a short final slice when one model attempt ends. | | windowChars | 1800 | Reasoning characters that force a background audit window. | | minWindowChars | 500 | Minimum pending reasoning for an opportunistic trigger. | | maxTraceChars | 12000 | Hard cap on the trace excerpt sent to one audit (tail preferred). | | cooldownMs | 1500 | Minimum spacing between background audits per session. | | requestTimeoutMs | 20000 | Deadline for a background session-less auditor call. | | turnStopTimeoutMs | 5000 | Maximum time agent/turn-stopping waits for a verdict. | | maxAuditsPerTurn | 8 | Maximum background audits one turn may start. |

auditor:

| Field | Default | Meaning | |---|---|---| | provider, model | null | Explicit route override; null follows the session's latest logged request/header, then the agent's creation options. | | maxTokens | 1600 | Auditor output-token cap. | | temperature | 0 | Greedy decoding. | | followReasoningEffort | true | Reuse the session's reasoning-effort selection when the route advertises one. | | maxResponseChars | 50000 | Cap on the decoded response before JSON parsing; over-cap fails open. |

evidence:

| Field | Default | Meaning | |---|---|---| | maxToolCalls | 120 | Tool calls retained per session. | | maxResultChars | 6000 | Per-result text retained in the ledger. | | maxDigestChars | 16000 | Overall evidence digest cap sent to the auditor. | | maxUserChars | 4000 | Per-user-message cap in the digest. |

gates:

| Field | Default | Meaning | |---|---|---| | maxTier | verify | Highest actuator tier permitted: observe, verify, block, cancel. | | minimumSeverity | medium | Findings below this severity remain audited but never drive a gate. | | verifyOn | unsupported | Lowest deterministic status that makes T1 force an extra step. | | blockOn | confirmed | What T2 treats as sufficient: never, contradicted, or confirmed. | | cancelOn | never | T3 evidence requirement: never or confirmed. | | consecutive | 2 | Windows at verify+ for T1; block needs this many; cancel needs at least two. | | t2Mode | ask | ask routes through the approval seam; deny blocks outright. | | askOnTimeoutForIrreversible | true | If an irreversible action races an in-flight audit, ask instead of allowing. |

store:

| Field | Default | Meaning | |---|---|---| | persist | true | Append verdict/action records to audit.jsonl in the audit dir. | | dir | null | Explicit audit directory; null derives from dshHomePath('thinking-auditor'), $DSH_HOME/thinking-auditor, or ~/.dsh/thinking-auditor. | | maxRecordsPerSession | 200 | Verdict/action records retained per session. | | maxSessions | 200 | Sessions retained in memory. |

Actuator tiers

• T0 — observe (always on). The verdict is stored and exposed through ctx.thinkingAudit; nothing acts.
• T1 — force verification (default maximum). At agent/turn-stopping, if the trailing history reaches verify, the plugin steers one neutral message asking the model to re-check each factual claim against a tool result or file read from the session. It never says why and it fires at most once per turn. This is the reference design's false-positive killer: most unbacked claims resolve here.
• T2 — block the action, not the thought (opt-in). At tools/pre-execute, if the verdict reaches block, the next action is irreversible (push/publish/network write/recursive removal/activation/…) and hysteresis is satisfied, the plugin returns ask or deny. Reversible and read-only actions continue.
• T3 — cancel (opt-in, both switches). Requires maxTier: cancel and cancelOn: confirmed, a mechanically confirmed fabrication, and at least two trailing block windows. It calls agent.cancel({ kind: 'hook', reason }); irreversible actions already executed are not rolled back, so pair T3 with T2 rather than relying on cancellation alone.

Defaults are deliberately capped at T1. A single model opinion is never enough to stop an interaction, and the auditor route is not independent: same-model monitors share blind spots and self-preference leniency.

Model experience

T1 is the only default behavior the model can notice. The injected message is neutral:

Before finalizing: verify each factual claim against an actual tool result or file read from this session, cite the evidence for anything you assert, and remove or explicitly mark as unverified any claim you cannot support from this session's evidence. Do not state a result you have not verified here.

T2's ask/denial reason is equally neutral and carries no monitoring details. The plugin never blocks a thought, never rewrites a request, and never injects context into the model's context unless a gate fires.

Privacy and safety

• The audit store contains claim quotes, brief bases, and route metadata. It is written only locally, 0600, under the harness home; it is not part of the canonical session log and is not uploaded by the official DeepSeek route. Delete the file or set store.persist: false to keep records memory-only.
• The auditor call still sends the trace window and evidence excerpts to the same provider the session already used. It adds a second same-model request whose content may contain session material; treat its cost and exposure as you treat the session itself.
• Trace and evidence are treated as hostile: escaped, tagged, and bounded before prompt assembly. The auditor is instructed never to follow instructions found inside them. The response parser accepts only a balanced JSON object and normalizes every field.
• A trace that infers it is being monitored can change its behavior. Covertness buys measurement fidelity now, not a guarantee.
• The auditor model never calls tools; only deterministic code can act, and only through documented dsh seams.

Known limitations

• Reasoning routes only. A model/route that emits no reasoning-delta chunks produces no audits. The plugin is a no-op there.
• Claim extraction is heuristic. It is tuned to be conservative; the model auditor and the LLM pass cover richer paraphrase, but neither is proof. unsupported means "not in this session's evidence", not "false".
• Same-model correlation. The auditor shares architecture, tokenizer, and blind spots with the subject; it is not an independent judge. Strategic deception modelling a same-capability monitor can defeat it.
• CoT unfaithfulness. A clean trace is not evidence of a clean computation; a suspicious trace is not evidence of intent.
• Audit latency. agent/turn-stopping may wait up to turnStopTimeoutMs for a verdict. Lower it, or set auditOnTurnStop: false, if the deployment values latency over the final gate.
• Boundary latency is real. T1 adds one model step; T2 can ask for approval. That is the point of the design, not a bug.
• No session-log events. Verdicts are available through ctx.thinkingAudit and the JSONL store, not through the session surface except for synthetic T1/T2 messages.
• Windows are form-aware, not semantic. A window can contain multiple attempts or span a step boundary; the auditor is told to use only the ledger.
• Gates are per-agent. A parent agent's block verdict does not gate a subagent's irreversible action; each session is audited and gated independently.
• Every audit is a model call. On a reasoning-heavy session, window triggers can add a second same-route request per window. Raise windowChars/cooldownMs, lower maxAuditsPerTurn, or set auditOnToolResult/auditOnAttemptEnd to false to trade coverage for cost.

Public service

When mounted, the plugin provides ctx.thinkingAudit:

• status() — enabled state, tier, session/verdict counts, in-flight audits, audit-store path/error.
• latest(sessionId) — latest detached verdict record.
• verdicts(sessionId, limit?) / actions(sessionId, limit?) — bounded detached history.
• onRecord(callback) — subscribe to { kind: 'verdict' | 'action', record }; observer failures are contained.

Development

npm run peers       # link node_modules/@deepseek-ai to the installed dsh closure
npm test            # node --test (pure suites plus the mock-context integration suite)
npm run test:local  # peers + tests in one shot
npm run check       # syntax-check every ESM source
npm run pack:check  # npm pack --dry-run

Tests run against the real @deepseek-ai/* packages from the installed dsh closure for schema and message construction, and use a mock Cordis context for lifecycle/event orchestration. No model API calls are made.

Verified against dsh 0.1.6-alpha.1 on Node 26; the plugin itself targets Node >= 20.

Deployment (NixOS profile)

This repository carries a staged module for the NixOS dsh profile:

• packaging/nixos/dsh-thinking-auditor.nix — the fetchFromGitHub derivation.
• packaging/nixos/default.nix.snippet — the pkgs/default.nix callPackage line.
• packaging/nixos/agents.nix.snippet — the home/agents.nix copy block and profile-patch row.
• packaging/README.md — placement and build steps.

License

MIT. See LICENSE.