pith-log

Collect AI model conversation logs into one normalized, faithful data source.

pith-log discovers the conversation logs that CLI AI tools leave on your disk (Claude Code, Codex, …), reads them verbatim, and emits them as a single normalized shape — JSONL for machines, Markdown for humans. Point it at a folder and it keeps that folder in sync. Then feed the result into your knowledge base, notes system, or any downstream tool.

It's embed-first: a small CLI (pith-log, alias plog) and a library. It does one thing — collect and normalize — and hands structuring off to whatever consumes it. Part of the pith ecosystem (feeds e.g. pith-wiki).

locate  →  read  →  normalize  →  render  →  stdout  |  --out-dir
(find     (verbatim  (per-source   (jsonl/    (pipe to     (one file per
 logs)     bytes)     parsing)      md/raw)    a host)       session + sync.json)

⚠️ Security. Conversation logs can contain plaintext secrets (API keys, tokens, pasted file contents). pith-log passes data through faithfully and does not redact. It never sends anything over the network — everything is local — but don't blindly > public.md or commit its output.

Install

npm i -g pith-log        # provides `pith-log` and the short alias `plog`

Requires Node ≥ 20. v1 targets macOS / Linux (Windows path encoding for the tool stores isn't verified yet).

Or run from source without installing (see Development): npm run dev -- <args>.

Quickstart

# zero-install: export every AI tool's logs on this machine into ./ai-logs
npx pith-log collect --out-dir ./ai-logs

pith-log sources                          # what tools are detected, and how many sessions?
pith-log list                             # preview sessions across all detected sources
pith-log collect --latest 5 --format md   # peek: 5 most recent as Markdown → stdout
pith-log collect --out-dir ./kb           # export everything to ./kb (all sources, incremental)
pith-log collect --source codex           # narrow to one source

Re-run any collect and only new/changed conversations are emitted.

Concepts

Sources. Each AI tool is a collector. --source defaults to all — every source detected on this machine (run pith-log sources to see them). Narrow with --source claude-code (or codex). See Supported sources.

Output formats (--format):

| format | content | use it for | | ----------------- | --------------------------------------------- | ----------------------------------- | | jsonl (default) | one full Conversation object per line, lossless | programmatic consumers, big batches | | md | deterministic Markdown with stable frontmatter | reading, one-pipe ingest | | raw | verbatim source bytes | byte-level provenance |

Rule of thumb: JSONL for fidelity, Markdown for legibility. Markdown is lean by default — thinking blocks omitted, long tool_result truncated (--include-thinking / --full-tools to keep them).

Incremental sync. collect defaults to all sessions but emits only what's new or changed since the last run, tracked in a small cursor. A conversation still in progress is re-emitted whole when it grows; downstream should upsert on the stable id. --full ignores the cursor and re-emits everything.

• Streaming to stdout: cursor lives at .pith-log/sync.json (project-local, gitignored). Redirect with --state <path> (e.g. one cursor per destination).
• Writing to a folder (--out-dir): the cursor lives inside that folder as sync.json, so the folder is self-describing and each destination stays independent (no cross-destination surprises).

One file per session (--out-dir <dir>). Instead of streaming to stdout, write <dir>/<id>.<ext> per session plus a folder-scoped sync.json. Re-running against the same folder rewrites only conversations whose rendered content changed (content-diff dedup). Because ids are platform-prefixed (claude-code: / codex:), multiple sources can share one --out-dir without collision.

Command reference

pith-log sources                                   # list sources + detection status
pith-log list [--source <id|all>] [--project <path>]
pith-log collect [options]

collect options:

| flag | meaning | | ----------------------- | -------------------------------------------------------------- | | -s, --source <id> | collector to use, or all (default: all detected sources) | | -f, --format <fmt> | jsonl (default) · md · raw | | -o, --out-dir <dir> | write one file per session into <dir> (folder-scoped sync) | | --full | ignore the sync cursor / index; re-emit everything | | --session <id...> | collect specific session id(s) | | -p, --project <path> | filter by project/cwd | | --since <date> | only sessions at/after this ISO date | | --latest <n> | only the most recent N sessions | | --state <path> | cursor path for stdout mode (default .pith-log/sync.json) | | --include-thinking | keep thinking blocks in Markdown | | --full-tools | don't truncate tool_result in Markdown | | --strip-attachments | drop inlined attachment bytes |

collect writes data to stdout; progress and diagnostics go to stderr, so pipes stay clean. Exit code is 0 on full or partial success and non-zero only when everything located failed.

Supported sources

| --source | reads from | notes | | ------------ | -------------------------------------- | ----------------------------------------------------------- | | claude-code | ~/.claude/projects/**/*.jsonl | full fidelity incl. inlined image attachments | | codex | ~/.codex/sessions/**/rollout-*.jsonl | v1: title/git not read from Codex's SQLite; external images kept as references, not inlined | | all (default) | every source above that is detected | run pith-log sources to see what's present |

Library API

import { collect, collectToDir, toMarkdown, registerCollector } from 'pith-log';

// Stream normalized conversations
for await (const conv of collect({ source: 'claude-code' })) {
  const md = toMarkdown(conv);   // deterministic, no LLM
  await host.ingest(md);          // structuring is the host's job
}

// Or materialize a folder with content-diff incremental dedup
await collectToDir({ source: 'codex', outDir: './kb', format: 'md' });

// Add your own platform without touching core
registerCollector(myCollector);

Also exported: toJsonl, getCollector, listCollectors, the Conversation / Message / ContentBlock types, ConversationSchema (zod), and the Collector interface.

Extending: add a source

Implement the Collector interface and register it — core, renderers, cursor, and CLI never change:

interface Collector {
  id: string;
  displayName: string;
  detect(): Promise<boolean>;                      // is this tool present?
  locate(filter?): Promise<SessionRef[]>;          // discover sessions (+ mtime/size)
  read(ref): Promise<RawCapture>;                  // verbatim bytes
  normalize(raw): Conversation;                    // SYNC, pure — fixture-tested
}

normalize is a pure function (bytes in, Conversation out), which makes source parsing trivially testable with fixtures. See src/collectors/claude-code/ and src/collectors/codex/ for references.

The normalized model

Every source maps to one Conversation: id (<platform>:<session>), source, title, timestamps, cwd/project, model, usage, messages[], and diagnostics[]. Each message is a list of closed-union content blocks — text, thinking, tool_use, tool_result, image, file. Platform-specific fields survive under extra.<platform>.*; byte-exact provenance is --format raw. See DESIGN.md for the full rationale.

Development

npm run dev -- list      # run via tsx, no build
npm run build            # tsc → dist/
npm test                 # vitest (normalize snapshots + CLI end-to-end)
npm run typecheck

License

Apache-2.0