sensemaking

Search and query a directory of Markdown notes from the command line. sense indexes frontmatter, prose and links in a local database. It can also combine word matches with links and semantic similarity.

Results contain file paths, snippets and line ranges. A person or agent can inspect the relevant passages without loading every note. No server or daemon is required; the default CLI query performs the incremental preparation it needs.

Problem

Markdown notes accumulate: research, decisions, meeting notes, agent output. Past a few dozen, finding the ones relevant to what you're doing means grepping or reading whole folders into context. The structure that makes notes navigable (frontmatter, wikilinks, headings) is exactly what a query needs, but nothing exposes it as a query surface.

sense indexes all of it into a local database (SQLite by default, or the experimental DuckDB store, see Config). By default, CLI queries reconcile the capabilities they need before reading; --no-build instead reads the last completed indexed generation without scanning live files. Nothing has to be running.

Quick start

npm install -g sensemaking
cd your-notes && sense init
sense download          # optional prefetch; build or the first default vector search fetches it otherwise

Requires Node.js 22.20 or newer. The default store uses Node's built-in SQLite.

sense map                                        # orient: fields, hub notes, recent changes
sense search "revenue OR earnings" --k 10        # locate: words + links + meaning, one ranked list
sense peek notes/q3-report.md                    # structure: outline + links, before reading
sense sql "SELECT path FROM frontmatter WHERE has(tags, ?)" urgent

A search result identifies the matching note, the evidence used to rank it and the relevant line range:

path                 snippets                                  via    score   lines
pricing-decision.md  …«pricing» decision … renewal «price»…    match  0.0167  L4-7

The row is illustrative. Actual paths, snippets and scores depend on the notes and configured search signals.

What sense indexes

Every file becomes rows in these tables, plus whatever an enabled feature adds of its own:

| table | holds | for | |---|---|---| | frontmatter | one column per key, plus path, _mtime, _size, _rank, _parse_error | filtering | | content | title, summary, text, path | text search and ranking | | links | src, target as written, dst resolved (NULL = dead link, but see the skill: a link to an attachment can never resolve) | graph | | tags | path, tag; frontmatter and inline #tags merged and deduplicated, nested tags stored full | tag filters | | sections | heading, level, start_line, end_line, tokens estimate | structure | | preset_files | path, preset | which presets cover which files; sql --preset binds these as a scope table to join, since sql is otherwise index-wide |

Results are references (path, title, summary, snippets), never file contents. Reading happens afterward through the filesystem, scoped to the line ranges peek returns. This is the just-in-time context pattern: the agent holds lightweight identifiers and loads payloads only when needed.

map has fixed-size output, a search row is tens of tokens, and a peek stays flat however large the note is. What it saves over reading grows with the file; a small note is cheaper to read whole.

-- filter and search compose in one query
SELECT f.path, content.title, snippet(content, -1, '«', '»', '…', 10) AS snippet
FROM frontmatter f JOIN content ON content.path = f.path
WHERE f.status = 'active' AND content MATCH 'revenue'
ORDER BY bm25(content, 10.0, 5.0, 1.0) LIMIT 10

Commands

| command | does | |---|---| | build [--force] | incrementally update the derived index and prepare every configured search capability; --force recreates only .sense/ | | map | doc count, frontmatter field coverage, top hubs by link rank, recent changes; hub/recent limits use bytewise path order on ties | | search "<text>" [--preset name] [--include glob] [--exclude glob] [--no-exclude] [--where "<sql>"] [--k n] [--snippet-char-limit n] [--snippet-count-limit n] | words + links + vectors, one fused ranked list; via labels each row's evidence. --k bounds notes returned, --snippet-char-limit (default 80) each passage, --snippet-count-limit (default 1) passages per note | | peek <path> [--preset name] [--where "<sql>"] | frontmatter + heading outline ([L143-162, ~380t]) + links both ways (first 20 per list, each with its total) | | path <a> <b> [--max-depth n] [--preset name] [--where "<sql>"] | shortest link chain between two notes, or none within the bound | | related <note> [--k n] [--preset name] [--where "<sql>"] | notes similar in meaning that <note> does not yet link to; reads vectors, so semantic-search cost | | sql "<statement>" [params...] [--preset name] | ad-hoc SQL over all the tables; ? binds positional args. Index-wide by default; --preset binds the preset's paths as a scope table the statement joins | | <name> [params...] | run a query saved in the config; --list names them | | init | write a starter sense.config.json | | status | index location, doc count, per-preset coverage, watcher heartbeat | | download | prefetch the embedding model named in the config; build, watch, or a default CLI vector query fetches it when needed otherwise | | watch | keep the index warm in the background (optional; see watch coordination) |

Query commands use the config's "build": true default to update the index first and prepare only what that operation needs. Core map, peek, path and SQL work do not prepare vectors. Set "build": false for a manual-build or watch workflow, or add --no-build for one query. These read the last completed generation without scanning source files or repairing missing readiness; they fail with a sense build instruction when the requested capability is not ready. This disables Sense index maintenance, not database access or arbitrary SQL.

The completed generation stores the exact decoded source used for indexed snippets. A no-build query reads those stored sources and checks readiness for the capabilities it requests; it does not hydrate from newer live files. A default query incrementally scans the configured tree for the capabilities it needs; vector preparation is limited to its eligible scope. sense build prepares every configured capability, and sense watch keeps them prepared as changes arrive.

Explicit sense build and sense watch prepare the index regardless of the config's build default. Watch builds before reporting ready, then processes edits; a no-build query can read the previous generation while an edit is being processed. Library callers choose preparation separately at open: open(config) prepares every configured capability, and open(config, { build: false }) opens existing compatible state. The config's build setting controls CLI queries only. build(config, { force: true }) recreates only the derived index. Public search, mapTree, and peek calls on one retained Store serialize with one another. Await those calls before running raw SQL, starting a transaction, or closing the handle.

search runs one text through every engine its scope has: FTS5 word match (BM25-ranked, bare words AND-join, operators are yours on sqlite; on duckdb and turso the FTS5 operators are a named error, see Config), a personalized-PageRank walk over the link graph, and vector similarity, fused into one list. via labels each row's evidence (match, link, vector, combinations). Within the vector signal, candidates rank by the true cosine against the best-matching canonical chunk before similarity is rounded to three decimals for display; a zero-direction vector has similarity 0, exact cosine ties use bytewise path order, and equal-scoring chunks choose the earliest authored chunk. The public search list ranks the combined word, link, and vector candidates by fused reciprocal-rank score. lines points at the section that earned the row (a direct read range). A vector-only row means the search words don't appear in that note; it showed up because the model judged it semantically related. --preset picks a named settings bundle from the config, --where filters on frontmatter. --format json on any reporting command returns structured output, and --format csv writes the row-returning commands one row per line, for redirecting a large result to a file instead of into context; --version and --help do what they say.

Search rows carry snippets: string[]. Each passage is generated around the matched words, marked with «», and normally limited to 80 characters by default. A whole matched word is preserved, so a passage can exceed that limit when the word is longer. --snippet-count-limit returns more non-overlapping passages from a note, in document order. Link- and vector-only rows have snippets: [].

Lexical words are case- and accent-insensitive. SQLite and DuckDB use their native English stem tokenizers; Turso's native Tantivy index has no stem tokenizer in the supported release, so it applies the shared Porter normalization to a derived field before native indexing. In every store, run, running, and runs match the same authored notes, while authored bytes are never rewritten. Quoted phrases require adjacent words, with punctuation treated as a separator, and punctuation-only input returns no lexical rows. SQLite's native caret and NEAR(...) expressions over unspaced text use original FTS5 tokens rather than sidecar substring semantics; use an ordinary quoted search when substring findability matters, at the cost of positional filtering.

Config

sense init writes sense.config.json; discovery walks up from cwd like git (--config <path> overrides). A config normally indexes its own directory. Set optional top-level root to keep configuration and cache state elsewhere: relative roots resolve from the config directory, never from the invocation cwd. Globs and stored path values are relative to that root; .sense/ remains beside the config, so separate consumers can index one vault with separate presets, queries, and caches.

{
  "$schema": "https://unpkg.com/sensemaking/schema.json",
  "version": 6,
  "build": true,
  "presets": {
    "default": { "include": ["**/*.md"], "k": 10 },
    "raw":     { "include": ["raw/**/*.md"], "k": 5 }
  },
  "embed": { "model": "minishlab/potion-retrieval-32M", "provider": "static" },
  "queries": {
    "dead-links": { "sql": "SELECT src, target FROM links WHERE dst IS NULL" },
    "by-tag":     { "sql": "SELECT path, title FROM frontmatter WHERE has(tags, ?) ORDER BY path" },
    "hot":        { "search": "pricing OR billing", "preset": "raw" }
  }
}

| key | holds | |---|---| | build | CLI query-time build default, true. Set false to query the existing index maintained by explicit build or watch; --no-build overrides true for one query. | | root | optional markdown-tree path. Relative to the config directory; omitted means that directory. Preset globs, filesystem reads, watcher events, and indexed path values use this root. .sense/ state stays beside the config. Changing it rebuilds the index. | | presets | named bundles of include/exclude globs, k (result count), signals (which engines this scope searches with, words, links, vectors; every signal whose prerequisites hold, unless the preset lists them exhaustively), where (a standing SQL filter). A file is indexed if any preset includes it, embedded if a model is named and some covering preset's signals include vectors; status shows each preset's coverage. | | embed | the model vectors are built with. Naming one gives the tree vectors; omitting the block means none at all, whatever the presets say. sense download fetches it. | | store | sqlite by default, or the experimental duckdb and turso. Each engine uses its own cache file, so changing this setting rebuilds the index. | | queries | entries runnable as sense <name>, each naming the verb it runs: { sql } for SQL (? binds positional args) or { search } for a ranked search with its settings baked in, so sense hot needs no flags. Running an entry validates it: a typo'd column errors and exits nonzero, and a parameterised entry validates with any argument, since preparing precedes binding. | | version | schema version; older configs auto-migrate on load, noted on stderr. |

Bare commands use the default preset; --preset names another; flags override single fields. Editing a preset rebuilds the cache and says which preset caused it.

Store choice

Choose sqlite for the smallest setup, full FTS5 query syntax, and concurrent Sense commands. Choose duckdb when the cache should participate in DuckDB analytical work over large datasets or in local/cloud workflows. Choose turso for its embedded Rust engine and Tantivy text index. The DuckDB and Turso adapters are experimental, install their native package on first use, and currently serialize Sense commands that open the same cache.

The commands and table names are shared. Raw SQL still follows the selected engine's dialect, and advanced FTS5 operators only work on sqlite. DuckDB and Turso hold their cache file for the life of a native handle, so Sense commands opening the same cache wait for the current command or watcher cycle to close; this is an adapter limitation, not a reader/writer coexistence guarantee.

Vectors need a model. Naming a Hugging Face id in embed.model is consent to fetch it when sense build, sense watch, or a default CLI vector query first prepares vectors, with progress on stderr, into ~/.sense/models (huggingface_hub's cache layout, one snapshot directory per resolved revision, shared by every tree, 124 MB, never in the package). sense download prefetches the same model ahead of time; it is idempotent and prints the resolved revision. embed.model is a Hugging Face id, or a path to a directory holding model.safetensors and tokenizer.json, which nothing fetches for you. A preset that asks for vectors when a local model path is missing those files is an error naming the fix, rather than a quieter result that would make the same search answer differently before and after; a preset whose signals exclude vectors never asks, so it is unaffected. An optional top-level "embed": { "model", "provider", "url", "key" } block points at any Model2Vec model, local path, or OpenAI-compatible endpoint (Ollama, LM Studio, hosted). Explicit builds and watch prepare all configured vectors. A default CLI vector query prepares only its eligible scope; --no-build requires that scope to be ready and never prepares it. Unrelated pending documents do not block either path.

Sense adds has(field, value) for array membership or string containment and basename(path) for path queries on every store. SQLite and DuckDB also provide segment(terms) for hand-written matching over text without word spaces. A frontmatter syntax error records _parse_error and leaves that file's discovered fields empty; the file's content still enters the index.

Providers

embed.provider picks the wire protocol; embed.model names the model. The verified integrations record what has actually been run against this codebase. Only those integrations are named as recommendations.

static. A local, pure-JS Model2Vec model; no network at query time.

"embed": { "model": "minishlab/potion-retrieval-32M", "provider": "static" }

model is a Hugging Face id, fetched to ~/.sense/models on first use, or a path to a local directory holding model.safetensors and tokenizer.json.

openai. Any endpoint serving an OpenAI-shaped POST /embeddings.

"embed": { "model": "nomic-embed-text", "provider": "openai", "url": "http://localhost:11434/v1" }

Ollama serves this shape at http://localhost:11434/v1. LM Studio serves the same shape at http://localhost:1234/v1, with narrower platform support than Ollama: Apple Silicon only on Mac, AVX2 on Windows. Content stays on the machine for either. Any other OpenAI-shaped endpoint, including a hosted one, works the same way through provider: "openai" and its own url; a hosted endpoint means tree content is sent to that service.

cohere. Cohere's native /v2/embed, which expresses the doc/query distinction (input_type) the OpenAI shape cannot.

"embed": { "model": "embed-v4.0", "provider": "cohere", "key": "COHERE_API_KEY" }

key names the environment variable holding the API key; the key value itself never goes in the config. Content is sent to Cohere.

Scale

Default CLI queries start with an incremental build of the capabilities they need; only changed files are re-parsed. --no-build skips that scan and reads the last completed generation. What to expect as a tree grows:

• Build work is linear in note count. Crawl and reconcile are the floor cost of a default query and the first thing to watch on a large tree; --no-build avoids them when snapshot semantics are appropriate.
• Output is flat. map, peek, and a search row cost the same on a small tree as a large one: context cost is bounded by what you ask for, not by how much there is.
• Bulk changes are paid by the next builder. That is normally the next default query. sense watch moves the re-parse into the background (watch coordination); --no-build continues to read the last completed generation. Run sense build --force to recreate the derived index.

For AI agents

npx skills add kmalakoff/sensemaking   # -g for global, -a claude-code to target

Three skills: sense for querying a tree, including store-specific SQL and search guidance; sense-setup for creating one and choosing its store, presets, vectors, and note conventions; and sense-bases for translating an Obsidian Bases .base file into sense SQL.

Prior art

• Effective context engineering for AI agents (Anthropic): agents should hold lightweight identifiers (file paths, links) and load payloads just in time, because context is a finite resource. The commands implement that pattern as a CLI.
• llm-wiki (Karpathy): an agent-maintained wiki navigated by an index.md and links, which he notes needs real search infrastructure past a few hundred pages. sense map derives that index from the notes instead of maintaining it; sense search is the hybrid local search it calls for.
• Agent memory patterns (llm-wiki's raw/wiki split, Claude Code's dreaming-style nightly consolidation) are trees of small notes with metadata, links, and layers of differing authority. sense is the query layer such patterns need: filter by metadata and age, scope by layer, surface near-duplicates semantically. It isn't an implementation of any one of them.

Alternatives

• Obsidian Bases/Dataview: same filters, but only inside the running app; agents can't query it headless.
• Index-on-build tools (MarkdownDB): query a snapshot; sense defaults to an incremental build before querying and also offers explicit snapshot reads with --no-build.
• Note CLIs (zk): fixed schema; sense filters on arbitrary frontmatter.
• Graph/LSP tools (IWE): structural queries over a markdown graph via LSP/CLI/MCP, retrieval by structure rather than similarity; no SQL, no vector search.
• Markdown vector stores (markdown-vdb): hybrid BM25 + vector search over markdown files, no frontmatter filtering; sense treats vectors as one signal alongside SQL, not the whole store.
• RAG / vector stores: similarity can't express WHERE status = 'active'. Here vectors are one signal inside search: same database file, filters compose, every row labels its evidence (via), and a preset turns vectors off per layer of the tree. No second store, no daemon, no native builds on the default store.
• Document-OS apps (Anytype, Logseq, SilverBullet, Capacities): full applications with their own UI and storage. sense is headless: your files stay files, there's no app to run.

Dependencies, all pure JS. No native builds by default.

| | | |---|---| | yaml | frontmatter | | markdown-it | markdown parsing | | @huggingface/tokenizers | chunking | | franc-min | language detection | | tinypool | worker pool for parallel parsing on large trees | | install-module-linked | installs the optional duckdb and turso bindings on first use, instead of shipping them to every install | | Node's built-in SQLite | the default store |

Plus two plugins for the GFM constructs the default preset lacks; tables, strikethrough, and autolinks are built in:

• markdown-it-footnote
• markdown-it-task-lists

License

MIT © Kevin Malakoff