pi-source-drafts

Persistent draft storage for information discovered during pi coding sessions.

Web searches, fetched pages, GitHub analysis, and user-provided external context are ephemeral — they vanish on session end or compaction. This extension preserves them as structured Markdown draft files in ~/.pi/source-drafts/.

Quick Start

npm install pi-source-drafts
# or
pi install npm:pi-source-drafts

# dev clone
git clone https://github.com/cframe1337/pi-source-drafts ~/.pi/agent/extensions/pi-source-drafts

Reload extensions (/reload in pi) or run a single session:

pi -e ~/.pi/agent/extensions/pi-source-drafts/src/index.ts

What It Does

| Trigger | Action | |---------|--------| | Tool call (captured via tool_result hook) | Action | |------------|--------| | web_search | Auto-captures query + results as a draft | | fetch_content | Auto-captures URL + content as a draft | | ctx_execute / ctx_execute_file / ctx_batch_execute | Auto-captures execution output | | ctx_index | Auto-captures indexed content | | LLM calls save_draft(...) | Saves structured info (by source type) | | LLM calls search_drafts(...) | Searches saved drafts with relevance ranking | | LLM calls draft_bundle(...) | Combines multiple drafts into a research brief | | User runs /save-source | Creates a manual draft | | User runs /drafts | Lists, views, searches, exports, or deletes drafts |

Commands

/drafts                        List all drafts
/drafts view <id>              Show draft content
/drafts search <query>         Find drafts by title/content
/drafts delete <id>            Remove a draft
/drafts export <id>            Export as standalone markdown
/drafts bundle <name> <ids>    Combine drafts into a research brief
/drafts compact                Write index snapshot
/drafts compact-content        Compact the binary content store
/drafts cc                     Alias for compact-content
/drafts stats                  Show statistics by source type
/save-source <title> | <text>  Save external info manually

LLM Tools

| Tool | Description | |------|-------------| | save_draft(title, content, sourceType, sourceUrl?, tags?) | Save structured information as a persistent draft | | search_drafts(query, sourceType?, tags?, limit?) | Search saved drafts by keywords, type, or tags — current project and session results rank higher | | draft_bundle(title, draftIds) | Combine multiple related drafts into one research brief |

Storage Layout (v0.2+)

~/.pi/source-drafts/
├── journal.jsonl           # Append-only WAL (single source of truth)
├── index.snapshot          # Binary checkpoint (fast startup)
├── index.format            # Format version marker
├── drafts.cdb              # Binary content store (optional, hot cache)
└── src-web_search-.../
    ├── draft.md            # Structured Markdown content (backward compat)
    └── meta.json           # Machine metadata

v0.1 auto-migrates on first run — old index.json + search.idx are converted to journal.jsonl and deleted.

Search Architecture (v0.2)

Inverted word index held in RAM — zero disk I/O for search queries. TF-IDF ranking with per-term document frequency + log-scaled term frequency. Titles weighted 2×, body 1×. Current project (+5) and session (+10) boost.

v0.1 vs v0.2 performance (benchmarked):

| Operation | v0.1 (200 docs) | v0.2 (200 docs) | |-----------|-----------------|-----------------| | Save | ~60 ms (rewrites whole search.idx) | 1.3 ms (append journal only) | | Search "deep dive" | ~5 ms | 3-5 ms (no disk read) | | Search "JavaScript" | ~3 ms | 3 ms | | Delete all 200 | ~5-15 s (rm -rf per folder) | 240 ms (journal append + memory) | | Startup cold | ~100 ms (parse JSON idx) | 34 ms (snapshot + short replay) |

No more JSON index rewrite. No more slow deletes. No more startup lag.

Delete is instant: one {"op":"delete"} line appended to the journal + removed from memory index. File cleanup is background.

Source Types

| Type | Description | |------|-------------| | web_search | Search engine query + results | | fetch_content | Fetched URL content | | user_source | User-provided external reference | | user_news | User-provided update/note | | code_snippet | Code example or pattern from analysis |

Features

• Auto-capture — web searches, fetched pages, and context-mode tool results are saved automatically
• Content de-duplication — FNV-1a hash prevents duplicate saves
• Section-level indexing — ## headings are indexed separately for precise search results
• Project/session context — each draft records the project directory and pi session, search prioritises current context
• Model tracking — each draft records the LLM provider/model used during capture
• Secret redaction — API keys, tokens, private keys, SSN, credit cards, phone numbers are automatically stripped before writing
• Draft bundling — combine related drafts into a single research brief
• Export — single drafts can be exported as standalone markdown with YAML frontmatter
• Binary content store — optional fast content cache (drafts.cdb), append-only with tombstone delete and in-place compaction
• Write queue — FIFO serialization for concurrent writes, reads are lock-free

Security

Credentials are automatically redacted before writing:

• API keys (sk-..., pk-..., rk-...)
• AWS keys (AKIA..., ASIA...)
• Private keys (RSA, EC, OpenSSH, DSA)
• GitHub tokens (github_pat_..., ghp_...)
• Bearer tokens
• SSN (\d{3}-\d{2}-\d{4})
• Credit cards (major formats)
• Phone numbers (CIS + US)
• Generic api_key, secret, password, token patterns

Drafts never leave your machine — they are stored locally only.

Development

bun test              # 71 tests, ~3s
bun test --timeout 120 # includes benchmarks

Project structure:

src/
├── index.ts              # Extension entry: hooks, tools, commands
├── draft-store.ts        # Façade: Journal + MemoryIndex + ContentStore + WriteQueue
├── rw-queue.ts           # FIFO write queue
├── journal.ts            # Append-only WAL + snapshot
├── memory-index.ts       # In-memory inverted index + TF-IDF
├── content-store.ts      # Binary content store (CDB)
├── scanner.ts            # Security scanner v2
├── migration.ts          # v0.1 → v0.2 converter
├── *.test.ts             # Tests + benchmarks
skills/                   # Skill definitions
ARCHITECTURE.md           # Full design document

License

MIT