pi-llama-slot-isolation (pi extension)

Prevents concurrent pi sessions from clobbering each other's llama.cpp KV cache by intelligently assigning each session to the best available slot.

The Problem

llama.cpp manages parallel inference through slots — independent KV cache partitions. By default (-np 1), there's only one slot. With multiple slots (-np N), the server uses a similarity heuristic (-sps, default 0.5 = 50% prefix match) to decide whether a new request can reuse an existing slot's cached context.

This breaks badly with pi:

• Multiple windows open on the same project send requests with different prompts (different turn, different tool calls). The similarity check fails, and llama.cpp assigns the new request to the last active slot, evicting the previous session's entire KV cache.
• Subagents spawn with fresh context. Their first request looks nothing like the parent's, so it gets routed to a slot that wipes the parent's cache.
• Result: every session re-processes its full context from scratch on every turn. Token throughput collapses. Latency spikes.

The Fix: Smart Slot Assignment

llama.cpp accepts id_slot in the request body of both /completion and /v1/chat/completions. When set, it pins the request to a specific slot, bypassing the similarity heuristic entirely.

This extension goes further than simple pinning: it queries the server's /slots endpoint to see how many tokens each slot has cached, then assigns the new session to the slot with the least cached tokens — minimizing damage to active sessions.

Slot Selection Logic

When a new session starts, the extension picks a slot using this priority:

1. Empty or abandoned — slot has no active claim (TTL expired, 120s). Among these, pick the one with the least cached tokens.
2. Coldest — among actively claimed slots, pick the one that hasn't been used longest. Spreads parallel requests across slots.
3. Lowest cache usage — if tied on recency, pick the one holding the fewest n_prompt_tokens.
4. Lowest slot ID — final deterministic tiebreak.

Summary: prefer empty > cold > low tokens > low id.

This logic lives in the compareSlots function and can be swapped out for weighted scoring later.

Cross-Process Coordination

Multiple pi windows are separate Node.js processes. The extension coordinates them through:

• State file (~/.pi/agent/llama-slot-state.json) — tracks lastUsed timestamp per slot per server. No claims, no ownership — just "who touched this last".
• File lock (.lock file) — async lock with proper event-loop yield, ensures atomic read-modify-write when acquiring slots. Re-entrancy guard prevents deadlocks.
• TTL-based expiry (120s) — slots not touched in 120s are considered free. No explicit release needed; crashes are handled automatically.
• Atomic writes — state is written to a temp file then renamed, preventing partial writes from corrupting the state.

Subagent Behavior

Subagents share the parent session's sessionId, so they inherit the same slot. This is intentional — they're part of the same conversation tree and benefit from the parent's cached context.

llama.cpp Slot Architecture

| Concept | Detail | |---------|--------| | -np N | Number of parallel slots. Each gets its own KV cache partition. | | Context split | Without -kvu, total context (-c) is divided evenly: each slot gets n_ctx / np tokens. | | -sps (similarity threshold) | Default 0.5. New request must share ≥50% prefix with a slot to reuse it. | | -kvu (unified KV cache) | All slots share one KV cache. Full context window per slot. Risk: if memory is exhausted, all slots may be purged at once. | | GET /slots | Returns an array of per-slot state objects including n_prompt_tokens (cached tokens). Used by this extension. | | Idle slot eviction | When memory is tight, idle slots are purged first, then active ones. |

Recommended llama.cpp Flags

llama-server -m your-model.gguf \
  -np 4 \        # enough slots for concurrent sessions
  -sps 0.0 \     # disable auto-matching (let id_slot control routing)
  -c 128000 \    # total context (split across slots unless -kvu)
  # -kvu \       # optional: unified cache if you have enough VRAM
  ...

Install

The easiest way — from npm (pi.dev gallery):

pi install npm:pi-llama-slot-isolation

The manual copy flow in SYNC.md still works if you prefer working from source.

Setup

1. Start llama.cpp with enough slots

llama-server -m your-model.gguf -np 4 -sps 0.0 ...

• -np 4 (or more): must be ≥ your maximum concurrent sessions.
• -sps 0.0: critical. Without this, llama.cpp's similarity heuristic can still override id_slot.

2. Environment variables (optional)

| Variable | Default | Description | |----------|---------|-------------| | LLAMA_SLOT_MAX | 4 | Fallback max slot count (used when /slots is unreachable). | | LLAMA_SLOT_PROVIDERS | (auto-detect localhost/127.0.0.1) | Comma-separated provider names, e.g. llamacpp,llamacpp2. | | LLAMA_SERVER_ROOT | (derived from baseUrl) | Override the server root URL for /slots queries. |

3. Sync the extension

Copy index.ts to the Pi extensions directory:

copy index.ts "%USERPROFILE%\.pi\agent\extensions\llama-slot-isolation\index.ts"

4. Reload pi

/reload

Important: After any source changes, repeat the copy + reload. Without syncing, Pi runs the old extension and tool contracts will be stale.

How It Works

First request (slot acquisition)

1. Hooks before_provider_request — fires after pi builds the provider payload
2. Checks if the target provider is a local llama.cpp server (proper hostname parsing via new URL())
3. Queries GET /slots to get live n_prompt_tokens for each slot (~42ms on idle server)
4. Acquires async file lock (cross-process, with re-entrancy guard)
5. Reads lastUsed timestamps for each slot
6. Sorts slots: empty > cold > low tokens > low id
7. Marks best slot with lastUsed = now, writes state atomically (temp + rename), releases lock
8. Injects id_slot into the request

Subsequent requests

Uses the cached slot from step 1 — no network, no disk, no lock. A coalesced timer updates lastUsed at most once every 2 seconds, batching rapid successive requests into a single write.

Session shutdown

On session_shutdown, the extension clears the slot cache for the current session and flushes any pending timers. The 120s TTL then handles the rest — if a session stops using a slot, it becomes stale and available to new sessions.

Files

| File | Purpose | |------|---------| | ~/.pi/agent/llama-slot-state.json | Shared state: lastUsed timestamp per slot per server | | ~/.pi/agent/llama-slot-state.json.lock | Atomic lock for concurrent access |

Multiple llama.cpp Servers

If you run multiple llama.cpp instances (e.g., different models on different ports), each gets its own slot space. The extension applies independently per server — session A might use slot 0 on server 1 and slot 2 on server 2.

Troubleshooting

Sessions still re-processing context

1. Check -sps 0.0 — most common cause. Without it, llama.cpp's similarity heuristic overrides id_slot.
2. Check state file — cat ~/.pi/agent/llama-slot-state.json shows current claims.
3. Check slot assignments — curl http://127.0.0.1:5001/slots | jq '.[] | {id, n_prompt_tokens}'
4. Verify /slots is reachable — if the extension can't reach the server, it falls back to hash-based assignment (less optimal).

Context window too small per slot

With -np 4 and -c 128000, each slot gets 32,000 tokens. If conversations exceed this:

• Increase -c (total context), or
• Use -kvu (unified cache, full context per slot), or
• Reduce -np if you don't need as many concurrent sessions

Slot collision (two sessions on same slot)

This can happen when all slots are active and none are stale. The extension will pick the slot with the lowest n_prompt_tokens — the least damage option. To prevent this, increase -np.

First request is slow

The first request for a new session includes the /slots query and file lock. On an idle server this adds ~42ms. Subsequent requests use the cached slot and are essentially free. If the server is unreachable, there's a 3s timeout before falling back to hash-based assignment.

Debugging slot assignments

llama.cpp includes id_slot in its response body. Use the provider-payload example extension to log request/response pairs and verify slot assignments.

Alternative Approaches

Unified KV Cache (-kvu)

All slots share one KV cache. Sessions can reuse cached prefixes from each other. Best when sessions share common system prompts or project context.

Trade-off: aggressive memory management. If VRAM fills up, all slots may be purged simultaneously rather than gracefully evicting idle ones.

Router Mode

llama.cpp's experimental router mode can proxy to multiple backend instances. Not recommended for this use case — it's designed for model selection, not session isolation.

Per-session llama.cpp instances

Run a separate llama-server per pi window. Maximum isolation, maximum VRAM usage. Overkill for most setups.