pi-l1-cache

L1 In-Memory Cache Extension for pi — CPU/RAM optimized, production-ready

A high-performance, production-ready L1 (in-memory) cache extension for the pi coding agent. Designed for stoic Unix simplicity: one file, one purpose, zero dependencies.

Features

| Feature | Description | |---------|-------------| | ~138µs overhead | End-to-end per-request cost (measured, incl. key hashing + JSON) | | Fast key hash | FNV-1a (~1.5µs/op) — no per-request CPU probe on the hot path | | Memory cap | Hard limit (20MB default) prevents RAM bloat | | Auto-eviction | LRU-style cleanup when limits reached | | CPU-aware | Auto-disables when CPU > 95% (checked once at startup) | | TTL-based | 1-hour default expiry for cached entries | | Atomic cleanup | Periodic expired entry removal |

Architecture

pi → [L1: RAM Map] → [L2: Redis via LiteLLM] → Provider
     ~138µs          ~150ms                   1-3s

Installation

# From npm (recommended)
pi install npm:pi-l1-cache

# From GitHub
pi install git:github.com/tobias-weiss-ai-xr/pi-l1-cache@main

# From local clone
pi install /path/to/pi-l1-cache

Then restart pi — the extension auto-loads.

Usage

# Show cache stats
/l1-cache

# Show detailed stats
/l1-cache stats

# Clear cache
/l1-cache clear

# Enable cache (if disabled)
/l1-cache enable

# Disable cache (if enabled)
/l1-cache disable

Stats Output

L1 cache status: ENABLED
Entries: 47 / 200
Memory: 4.2MB / 20MB
TTL: 3600s | CPU threshold: 95%
Hits: 23 | Misses: 70 | Evictions: 5
Hit rate: 24.7%
Init CPU: 45.2% (ok)
Last cleanup: 2025-08-22T10:30:00.000Z

Configuration

Environment Variables

Change defaults without modifying source code:

| Variable | Default | Description | |----------|---------|-------------| | L1_CACHE_ENABLED | true | Master enable/disable | | L1_CACHE_MAX_ENTRIES | 200 | Maximum number of cache entries | | L1_CACHE_MAX_MB | 20 | Maximum memory in MB | | L1_CACHE_TTL | 3600 | TTL in seconds (1 hour) | | L1_CACHE_LOG | false | Enable debug logging |

Example:

# Disable cache
L1_CACHE_ENABLED=false pi

# Use 50MB cache with 30-minute TTL
L1_CACHE_MAX_MB=50 L1_CACHE_TTL=1800 pi

Default Settings

Edit src/index.ts (lines 30-38) to change compiled-in defaults:

const DEFAULTS: Settings = {
  enabled: true,
  maxEntries: 200,
  maxMemoryBytes: 20 * 1024 * 1024, // 20MB
  ttlSeconds: 3600,                // 1 hour
  cpuThreshold: 95,                 // disable if CPU > 95%
  logStats: false,
}

Design Philosophy

Stoic Unix Principles

• One thing, done well — Caching, and only caching
• Do not rely on external services — Pure in-memory, no Redis
• Graceful degradation — Works even on constrained systems
• Zero dependencies — Single TypeScript file

Performance Optimizations

1. Cheap key hashing (FNV-1a, ~1.5µs/op). Note: Node's native SHA-256 (OpenSSL) is marginally faster than a JS FNV loop — hashing is a rounding error next to JSON.stringify(messages), and both are >1000× below provider latency. The real win is not a faster hash, it is skipping the provider call.
2. L1 only — avoid disk I/O in hot path
3. Batch eviction — remove 10-20% at a time, not one-by-one
4. Periodic cleanup — async, non-blocking garbage collection
5. Single CPU check — at startup only, not per-request

Testing

# Run all tests
npm test

# Watch mode
npm run test:watch

# Check TypeScript
npx tsc --noEmit

17 tests covering:

• Hash consistency & collision resistance
• Size estimation for various data types
• LRU eviction behavior (entry count + memory)
• State management & reset
• Settings override

Benchmark

Measured against the published npm artifact ([email protected]) on Node 22, using the package's own test hooks and a mocked ExtensionAPI with a realistic 15-message conversation history:

| Measurement | Result | |---|---| | fastHash (FNV-1a) standalone | ~670k ops/s — 1.49µs/op | | Node native sha256 (for reference) | ~850k ops/s — 1.18µs/op | | Full put: hash + map set + size check | ~249k ops/s — 4.0µs/op | | Interceptor path (hit and miss, 15-msg history) | ~138µs/request | | Cache hit vs uncached provider round-trip (1–3s) | ~7,000–20,000× wall-clock | | Key uniqueness over 50k synthetic keys | 50,000/50,000 (no collisions) | | Eviction under 200-slot cap, 20k inserts | cap held; 19,800 evicted |

⚠ Correction: earlier versions claimed FNV-1a was "~100× faster than SHA256". That was wrong — Node's native SHA-256 is ~0.8× faster in practice. The hash was never the bottleneck: JSON.stringify dominates the ~138µs per-request cost. Cache hits are keyed by byte-identical requests, so real-world hit rate depends on your workload (best for retries, repeated tool calls and same-prompt reruns).

pi API compatibility

Verified against the pi 0.84.x extension API — two runtime facts shape the behaviour:

• before_provider_request receives the assembled provider request as event.payload (model, messages, parameters); cache keys are derived from it. In current pi this hook is a payload transform, not a response short-circuit, so a cached response is only ever returned once a response body has actually been captured.
• after_provider_response currently carries only { status, headers } — no response body. Until a pi version exposes the body, responses cannot be stored; the extension detects this, logs a one-time note, and keeps /l1-cache stats working. It upgrades automatically (no config) on any pi version that exposes the body.

On pi 0.84 the extension therefore operates as a request-key instrumentation layer (Hits/Misses/Evictions via /l1-cache, CPU guard, TTL bookkeeping) and only short-circuits identical requests when the API contract provides the body it needs.

Related Projects

• opencode-saia-plugin — SAIA provider for OpenCode
• zot-saia-plugin — SAIA provider for zot CLI
• pi-saia-plugin — SAIA provider for pi coding agent

License

MIT — see LICENSE

Maintainer

Tobias Weiß — [email protected]