dsh-boost-probe
v0.0.4
Published
Runtime evidence for DSH BOOST practices: load a published plugin in a throwaway DSH profile, observe effects/inject/dispose, emit JSON. Non-official; self-assessed.
Maintainers
Readme
dsh-boost-probe (MVP)
Runtime evidence for DSH BOOST practices — one command, any installed DSH, no plugin changes.
Non-official. No certification, no ranking, no
verified. Every result isself-assessed.
Why
DSH and cordis move fast. Asking every plugin author to change code for every release is not sustainable. This probe takes the opposite approach:
- The plugin stays as published. We only load it and observe.
- The host version is recorded, not locked. Evidence says which DSH/node it came from; a different host means the evidence does not transfer silently.
- Capability detection, not version gates. Missing hook →
not-run, not fail.
Quick start
node cli.mjs validate # 4 fixtures + one client-side case (no DSH needed)
node cli.mjs run dsh-fast # throwaway DSH_HOME, real boot, JSON evidence
node cli.mjs run dsh-community-bench # client-side: explicit not-run reasonrun creates a temporary DSH_HOME, installs the plugin with --ignore-scripts,
injects probe.mjs through a --patch overlay, waits for the target fiber to
settle, disposes it, and prints a summary. Nothing touches your real profile.
Options: --target <fiber> (default: package name without dsh-),
--profile <base|web|name> (default: one-off boostprobe), --restarts n
(also DSH_BOOST_PROBE_RESTARTS=n), --settle ms,
--timeout ms, --out file, --keep.
What it measures
| LEV | Question | Evidence |
|---|---|---|
| LEV-1 | Are effects registered and released? | effectsBeforeDispose / effectsAfterDispose, disposedUid, leakedTimersOwned (ALS) + leakedTimersWindow; each restart cycle also carries a post-dispose dispose sample |
| LEV-2 | Does inject wait instead of crashing? | targetState: pending → active (or failed), activeMs (legacy boot→active), applyNs / applyFromLoadingNs (hrtime, target-scoped), restarts[] (N × applyNs) |
| LEV-7 | Can the probe observe tool concurrency? | lev7SelfTest: 8 concurrent tools/execute calls via ctx.root.waterfall (30 ms next), only when the host exposes it; tools.calls / tools.peak include them. This is an instrument self-test, not a concurrency verdict on the plugin. |
| LEV-6 | What does the provider report for prompt/completion/cache tokens? | usage[] from the llm/stream waterfall, opt-in with DSH_BOOST_PROBE_USAGE=1; provider-native names and DeepSeek aliases. A synthetic: true row is an endpoint check, not real cache/billing evidence. |
lev7SelfTest runs after the target is ACTIVE (or before the watchdog writes
evidence). It is skipped with status: "not-run" when ctx.root.waterfall is
not a function. tools.pluginCalls excludes the eight self-test calls, so
plugin concurrency and instrument concurrency are not conflated.
Apply timing and restart loop
applyNs is measured with process.hrtime.bigint() (monotonic, ns) from the
target fiber's internal/plugin event to its first ACTIVE. Because an async
apply can wait on services, applyFromLoadingNs is also recorded: it starts
at LOADING (the plugin callback is about to run) and isolates the apply body
from plugin creation. activeMs is unchanged for compatibility: it is
Date.now() from the first plugin event of the boot to target ACTIVE, i.e.
boot latency, not apply time.
--restarts n (or DSH_BOOST_PROBE_RESTARTS=n) exercises the public cordis
fiber.restart(): it disposes the current instance and re-runs the same
module's apply in the same fiber, so the probe needs no module reference.
Each cycle is recorded in restarts[]:
applyNs—LOADING→ACTIVE(hrtime);unloadNs— restart call →LOADING, i.e. the dispose path;dispose.effectsAfterDispose/dispose.leakedTimersOwned— sampled atLOADING, after the previous instance's disposers ran and before the next apply starts.
The LEV-7 self-test is deferred until after the restart loop when restarts are
requested, so self-test timers never contaminate the disposal samples. If the
host fiber has no restart(), the mode is reported as unavailable and no
cycles are invented; a CLI-side boot loop is then the only alternative.
LEV-6 usage capture (opt-in)
DSH_BOOST_PROBE_USAGE=1 wraps the host's llm/stream waterfall and records
every terminal usage chunk in out.usage[]:
prompt_tokens,completion_tokens,total_tokens,reasoning_tokens;cache_read_input_tokens,cache_creation_input_tokens;- DeepSeek aliases
prompt_cache_hit_tokens/prompt_cache_miss_tokens.
prompt_tokens is reconstructed as inputTokens + cacheReadTokens +
cacheWriteTokens, because dsh-llm's TokenUsage.inputTokens already excludes
cache reads and writes; the raw adapter_usage is kept beside it. The capture
is provider-neutral and does not read keys or request bodies.
DSH_BOOST_PROBE_USAGE_REQUEST=1 makes the probe issue one request through the
real llm.stream() after the target is ACTIVE (..._USAGE_PROVIDER /
..._USAGE_MODEL, default synthetic-openai / synthetic-model). Without it,
the probe waits up to DSH_BOOST_PROBE_USAGE_WAIT_MS (default 60000) for the
host/agent to make its own request, then finishes. Finish is not scheduled
before the request settles, so evidence is written after capture.
Rows are marked synthetic only when DSH_BOOST_PROBE_SYNTHETIC=1. Use that
flag only with a local fake endpoint: a synthetic row proves the instrument
path, never a real provider cache hit or a billing claim.
For a real key, run ... --usage is the shorthand: it enables capture, makes
two calls to deepseek-official/deepseek-flash in one session, and records a
provider error (for example a missing key) in usageCapture.requestError.
DEEPSEEK_API_KEY=<key> node cli.mjs run dsh-fast --target fast --usage --out real-usage.jsonEnd-to-end check against a local OpenAI-compatible endpoint:
node verify-usage.mjs --target-tarball /tmp/boost-exp/arms/dsh-fast/boost.tgzIt starts fixtures/synthetic-openai.mjs on 127.0.0.1, installs the probe
and the target into a throwaway DSH_HOME, points llm-pi-ai at the endpoint,
and asserts the captured prompt_tokens / completion_tokens / cache fields
match the endpoint's counters.
Client-side bundles (dsh.client)
A package that declares dsh.client ships a browser-side bundle (a panel that
runs in the web client). The server-side probe cannot observe that code, no
matter which profile loads it. Rather than time out with a vague not-run:
cli.mjsreads the installed package'spackage.jsonafterplugin add; ifdsh.clientexists it passesDSH_BOOST_PROBE_CLIENT=1to the probe.- On watchdog timeout / target-never-appeared, the probe writes
result: "not-run",clientSide: true, andreason: "client-side bundle (dsh.client); the server-side probe cannot observe it". - The CLI summary carries
clientDeclared,clientPackage,clientSide, andreason, so the limitation is visible in the evidence itself.
[email protected] is such a package: on 2026-11-05 the base profile
returned exactly that not-run + client-side reason. This is a statement
about the probe's observation surface, not a claim that the package is not
a plugin.
leakedTimersOwned uses AsyncLocalStorage: timers created inside the target
fiber's effect callbacks are attributed to that fiber, so host timers are
filtered out. leakedTimersWindow is the old window-based count kept for
debugging; it includes host noise. The fixtures below prove the owned count
catches a real leak and stays empty for clean plugins.
Fixtures (negative controls)
| Fixture | Expected |
|---|---|
| leaky | disposer runs but forgets a timer → leakedTimersOwned non-empty |
| eager | dereferences a missing service → failed |
| waiter | needs a later-provided service → pending → active |
| restarter | 3 × fiber.restart(): every cycle has applyNs, effectsAfterDispose=0, leakedTimersOwned=[] |
| burst | 16 concurrent tools/execute → tools.pluginCalls=16, plus lev7SelfTest 8/8 |
If a fixture does not behave as expected, the corresponding LEV is not-run —
an instrument that cannot catch the negative control is not evidence.
Example (2026-11-05, dsh 0.2.1-rc.5)
[email protected] on this machine: result=disposed, activeMs 7.7–12.6s,
effectsBeforeDispose=6, effectsAfterDispose=0.
The six effects were released and no timer was left inside the plugin's own
effect callbacks (leakedTimersOwned=[]); the window count (2) was host noise.
LEV-7 self-test in the same run: status=run, calls=8, peak=8, failedCalls=0.
2026-11-06 pilot: [email protected], 5 in-host restarts
result=disposed, restart.mode=fiber.restart, completed=5. Initial
applyNs was 8.33 s (the plugin awaits ctx.storageDomain.open), so the
restart cycles are the comparable numbers: 5 cycles gave
min/median/max = 1.256 / 2.158 / 5.275 ms. Every cycle reported
effectsAfterDispose=0 and leakedTimersOwned=[]; the final dispose went
6 → 0 effects. This is a one-arm instrument pilot on one host, not an effect
estimate and not a BOOST/unboost comparison.
Evidence schema (draft)
{
"schema": "boost-runtime/1",
"target": "fast",
"host": { "dsh": "0.2.1-rc.5", "node": "v24.22.1" },
"result": "disposed",
"targetState": "disposed",
"applyNs": 8332163561,
"applyFromLoadingNs": 8331998978,
"activeMs": 13045,
"restarts": [
{
"index": 1,
"method": "fiber.restart",
"applyNs": 3908630,
"unloadNs": 452463,
"dispose": { "effectsAfterDispose": 0, "leakedTimersOwned": [] }
}
],
"restart": { "requested": 1, "mode": "fiber.restart", "completed": 1 },
"effectsBeforeDispose": 6,
"effectsAfterDispose": 0,
"leakedTimers": ["timeout", "timeout"],
"tools": { "calls": 8, "peak": 8, "pluginCalls": 0, "selfTestCalls": 8 },
"lev7SelfTest": { "status": "run", "calls": 8, "peak": 8 }
}Outcomes
| result | meaning |
|---|---|
| disposed | target reached ACTIVE, was disposed, evidence recorded |
| incompatible | DSH refused the install because the plugin's host range excludes this DSH; required lists the ranges |
| not-run | target never appeared (clientSide when dsh.client explains it), or timeout |
| failed | target fiber reached FAILED |
Limits
- Not a host boot proof for every profile; the throwaway profile is base + the
plugin under test.
--profile webadds the web-app layers but does not make client-side bundles observable. - A plugin declaring
dsh.clientis server-side invisible: the probe returnsnot-run+clientSide: true+ an explicit client-sidereason. Testing the browser panel would require injecting the probe into the web client. - LEV-7 needs the host's
tools/execute; absent →lev7SelfTest.status="not-run". When present, the eight-calllev7SelfTestproves the instrument sees concurrency; it is not a plugin concurrency conclusion. - Timer ownership covers effect callbacks; timers created directly in
applystill fall back to the window count. The probe's own self-test timers are excluded fromleakedTimersWindow. - No concurrency-capacity measurement (a bounded queue's size is not observable from outside).
