Sponge

Sponge is a local library for articles, images, PDFs and media transcripts. Capturing runs on your machine through Ghostget; saved content stays in a private SQLite database and adjacent hashed asset files.

Version 0.2.0 introduces the local-library CLI; earlier 0.1.x commands are inactive. The native desktop app and cloud sync are not available yet. The website is sponge.computer. Previous research lives at hraness/sponge-research.

CLI

Node.js 24.20 or newer is required. Capture currently targets Apple Silicon macOS; read and queue interfaces remain portable.

Install the released package with npm install -g @hraness/sponge. For development from this checkout:

npm ci
npm run build
node bin/sponge.js init
node bin/sponge.js setup --capture
node bin/sponge.js add https://example.com/article --wait
node bin/sponge.js list
node bin/sponge.js show ITEM
node bin/sponge.js search 'your query'

The default library is ~/.local/share/sponge. Use --library /absolute/path for another private library. add without --wait queues a link; run --drain processes queued links sequentially. refresh ITEM --wait attempts a new revision and preserves the old one if capture fails. --json provides structured output. Errors are categorical JSON on stderr; exit codes are 0 success, 1 failure, 2 invalid input, 3 partial content, 4 action required, and 130 cancelled.

Capture setup explicitly downloads pinned Ghostget, Wordcell and Bun artifacts plus frozen dependencies. Read/search never download runtimes or contact source websites. Setup reports disk requirements, a monitored 2 GiB temporary installation limit, a 1 GiB retained runtime limit and a 30-minute dependency installation deadline. Temporary dependency caches are removed before the runtime is promoted. doctor reports runtime integrity and pending cleanup. Private-network capture requires --allow-private-network with --wait or run --drain.

Images use sponge-asset:SHA256 references. asset ITEM SHA256 > image.png retrieves verified bytes offline; asset ITEM SHA256 --json returns its metadata. Unresolved remote images become unavailable markers, and captures retain partial status. delete ITEM hides content and removes it from search. queue status, queue cancel JOB, and queue retry JOB manage work. After an interruption, queue recover verifies that the old worker and its subprocesses have stopped before making work available again. Uncertain cleanup remains blocked.

PDFs and media

node bin/sponge.js add /absolute/path/paper.pdf --kind pdf --wait
node bin/sponge.js add https://example.com/episode --kind media --wait

PDF capture requires Poppler (pdfinfo and pdftohtml). Tesseract enables optional local OCR. Provider media acquisition requires yt-dlp; audio validation requires ffprobe from FFmpeg. Install these separately, for example with brew install poppler tesseract yt-dlp ffmpeg on macOS. Capture setup does not install system tools.

PDF capture retains the original and readable Markdown with page references. Wordcell owns extraction and reports uncertain layout. Media capture prefers available captions and retains timestamped transcripts, metadata and thumbnails. Temporary audio is deleted after transcription and verified cleanup.

Local inference requires Apple Silicon and macOS 26 or newer. Release packages include a manifest for optional native downloads; a source checkout without that manifest reports runtime-missing. Installation never downloads models.

Native helpers are distributed from the private hraness/sponge repository. For explicit native setup, provide GH_TOKEN (or GITHUB_TOKEN) with read access to that repository's contents. Sponge never saves the token and sends it only to GitHub's API; public model downloads do not receive it. With an authorized GitHub CLI session, for example: GH_TOKEN="$(gh auth token)" sponge setup --asr. Repeat with --embedding for search. Already verified caches work without a token. Installed helpers and models are identified by content, so an upgrade that ships byte-identical native artifacts keeps working without rerunning setup; a release that changes them reports runtime-missing until setup runs again.

sponge setup --asr
sponge add https://example.com/episode --kind media --wait --language en
sponge refresh ITEM --wait --language en --force-local
sponge setup --embedding
sponge index create
sponge index run
sponge index status
sponge search 'your query'

Local ASR currently supports English. --force-local bypasses provider captions. Language choices apply to the current --wait, run --drain, or worker serve invocation; queued-only submissions reject these flags. worker install --language en saves the English ASR choice for background jobs. Search combines local full-text and semantic results when a compatible index and model are available, and reports when semantic search is unavailable. --mode lexical uses only full-text search. Use index create --profile NAME and search QUERY --profile NAME to keep separate search profiles. The active index follows the library: each new capture, refresh or delete queues only its new chunks, which the worker (or index run) embeds, and identical chunks are embedded once. index status shows the active generation's coverage and what is pending; index status GENERATION shows one generation. Model changes create separate index generations; a new one becomes active only after it covers the current library. index rollback GENERATION requires a complete generation compatible with the current revisions. index prune --dry-run lists what pruning would delete; index prune deletes the coverage of cancelled, failed and retired generations, then the vectors no kept generation covers (including those of deleted items), keeping the active and any building generation plus the most recent rollback-ready one per profile (--keep N, 0-16). It refuses while index work, cleanup or a backup is running. Freed pages are reused; --vacuum also shrinks the file after checking free space. Rollback to a pruned generation needs a rebuild.

Digests

A digest (Digest v2) is a short gist led by one sentence and 3–5 ideas, each a claim with detail and 0–2 verbatim quotes, with 1–6 quotes in all. A local model proposes it; Sponge checks every rule against the captured text and stores only a passing draft. Drafting is opt-in and needs a model server you start yourself, such as llama.cpp llama-server, LM Studio or Ollama:

llama-server -m gemma-4-12b-it-Q4_K_M.gguf --port 8080
sponge digest ITEM --endpoint http://127.0.0.1:8080 --model NAME
sponge digest ITEM
sponge review DIGEST --accept all --reject quote:1.0

--model must match an id the server lists at /v1/models. Endpoints must be literal loopback HTTP (127.0.0.1 or [::1]); Sponge refuses anything else so captured text never leaves the machine, and never starts or downloads a model. Suggested models are Gemma 4 12B on 16 GB Macs and Gemma 4 E4B on 8 GB, with Qwen3.5 9B or 4B as alternatives. Rerunning with the same inputs returns the stored draft. Review targets are all, gist, idea:N and quote:N.M (0-based); only accepted fields can be published. --check-support also requires each quote to support its claim by local embedding similarity. See digest drafting.

Export, recovery and background work

sponge export ITEM --output /absolute/new/export-directory
sponge backup --output /absolute/new/library.sponge
sponge restore /absolute/library.sponge --into /absolute/new/library
sponge gc --dry-run
sponge gc --apply /absolute/path/to/report.json
sponge gc --runtimes --dry-run
sponge worker install --language en
sponge worker status
sponge worker uninstall

Exports contain Markdown, images and saved notes, never the publishing key. Backups preserve revisions, assets, reading state, notes and the publishing key, so a restored library publishes under the same identity; keep backups private. Restore requires a new directory, verifies every file and leaves connections unresolved; runtimes and credentials are excluded. Garbage collection removes only unreferenced blobs after rechecking the report. gc --runtimes removes capture runtimes and model packages this Sponge no longer pins, plus settled worker journals; --dry-run only lists them. It holds worker custody, keeps models an active or building index generation needs, and refuses symlinks and anything in use. Successful setup runs the same collection.

The background worker is opt-in and processes work sequentially. Its LaunchAgent pins the installed command and Node; after a Sponge or Node upgrade, worker status and doctor report the drift and sponge worker install reinstalls it in place. Its output goes to runtime/service/launchd.log in the library, rotated at 1 MiB. Uninstall stops the worker and preserves the library. In-session startup and cleanup are checked; fresh-login startup remains unqualified. See importing existing captures for a preview-first import flow.

Public reading list

sponge publish --dry-run
sponge publish --output /absolute/site --title 'My reads' --base-url https://reads.example.com/ --min-age-days 7

Items captured from public sources are public by default; visibility ITEM private hides one. publish --output writes a static site: a JSON Feed (feed.json, newest 100), monthly shards under items/, an Atom feed.xml, and an index.html that uses the bundled <sponge-list> element. An Ed25519 key created in the library on first publish signs a manifest of every file's hash; the private key stays in the library. The same library state gives byte-identical files. --min-age-days N holds back items listed in the last N days. Nothing is uploaded. Embed the list elsewhere with <sponge-list src="https://reads.example.com/feed.json" verify></sponge-list> and sponge-list.js; the host must allow cross-origin reads, and verify needs HTTPS. See the decision record.

Selected authenticated connections

The initial development interface accepts private connections.json in the library (mode 0600). It contains references to existing Ghostget accounts, never cookies or tokens. A future setup phase adds connection-management commands.

{"schemaVersion":1,"connections":[{"id":"personal","kind":"account","stateHome":"/absolute/ghostget-auth","accountId":"personal"}]}

Use add URL --connection personal --wait. A public capture never silently selects your browser or account. For articles, attached browser connections use kind: "cdp" and an explicitly selected local endpoint; attached browsers remain open. PDF and media capture support selected Ghostget accounts, not CDP. Remote PDF acquisition requires public network addresses; --allow-private-network is not supported for PDF capture. Live authenticated remote-PDF qualification remains pending; failures never fall back to a different account or a public request. Changing connection identity creates a distinct private source realm.

Design and validation

See the design, phase plan, and evidence inventory. PDF improvements belong upstream in Wordcell; Sponge owns library ingestion and offline reading.

npm run typecheck
npm test
npm run build
npm run verify:package

Use the repository's host scheduler for required local resource lanes. The installed capture qualification also requires a separately started controlled fixture server, its allocated PORT, and an installed SPONGE_INSTALLED_CLI: npm run verify:steel-thread -- --installed. It tests public/authenticated capture, restart, offline read/search/assets and failed refresh preservation.

The website lives in site/: npm ci, npm run dev, and npm run build there.

Releases

After a version bump reaches protected main with green CI, the release tagger creates an annotated version tag. Release CI builds, packs and installs the exact artifact, then publishes immutable GitHub Release assets. The same verified tarball is mirrored to npm through trusted publishing. No runtime installation runs as an npm lifecycle hook. The repository is private, so npm provenance is disabled. The website stays out of the package.