pi-inline-image-read

Auto-inline binary assets (images, audio) via a custom read_asset tool. Images attach directly to vision-capable models; audio is transcribed to text via a configured external model. Drop-in upgrade over the built-in read tool for binary files.

Part of the pi coding agent extension ecosystem.

Why

pi's built-in read tool reads text files. When the active model is multimodal (e.g. minimax-cn/MiniMax-M3, anthropic/claude-3.5-sonnet, openai/gpt-4o), the image can be passed to the model as an ImageContent block. Without this extension, the model never sees the image when the agent uses read. Forcing the user to manually drag-drop attachments is friction.

This extension:

1. Registers a new read_asset tool that LLM is guided to use for binary files.
2. For images: inlines the image directly into the next LLM request (zero roundtrips, no extra tokens).
3. For audio: POSTs the audio to a configured external model (currently minimax-cn/MiniMax-M3 works because M3 is multimodal) and injects the text description into the agent's tool result. Works with any active model.
4. Defensive: if the LLM uses the built-in read tool on a binary file, the call is blocked with a clear "use read_asset instead" message.

Comparison

| Approach | Tokens | Roundtrips | Native vision? | |---|---|---|---| | Plain read for image | 0 (text "image read" string) | — | ❌ model can't see it | | describe_image via vision model | input + output tokens | 1 extra | ✅ via text | | This extension (image inline) | image tokens only | 0 extra | ✅ native | | This extension (audio transcribe) | image-side input + output tokens | 1 extra | ❌ (text only) |

Install

# From a local checkout (no npm publish needed)
pi install /path/to/pi-inline-image-read

# From git
pi install git:github.com/chenty/[email protected]

# From npm (after publish)
pi install npm:[email protected]

Or try without installing:

pi -e /path/to/pi-inline-image-read

Configuration

Add to ~/.pi/agent/settings.json (global) or <project>/.pi/settings.json (project overrides global):

{
  "inlineImageRead": {
    "schemaVersion": 3,
    "images": {
      "enabled": true,
      "pipeline": "inline",
      "maxBytes": 5242880,
      "mimeTypes": ["image/png", "image/jpeg", "image/gif", "image/webp", "image/bmp"]
    },
    "audio": {
      "enabled": true,
      "pipeline": "transcribe",
      "maxBytes": 20971520,
      "mimeTypes": ["audio/wav", "audio/mpeg", "audio/mp4", "audio/ogg", "audio/flac"],
      "transcribe": {
        "provider": "minimax-cn",
        "model": "MiniMax-M3",
        "prompt": "Listen to the audio and describe in detail. Mention speech content if any, plus audio characteristics (length, tone, language, speakers). If silent, say so. Reply in plain text only — no markdown, no preamble.",
        "maxOutputTokens": 1024,
        "timeoutMs": 120000
      }
    }
  }
}

Per-mode fields

| Field | Type | Default | Notes | |---|---|---|---| | enabled | boolean | true (images) / true (audio) | Master switch for this mode. | | pipeline | "inline" | "transcribe" | inline (images) / transcribe (audio) | How to route this mode. | | maxBytes | number | 5 MB (images) / 20 MB (audio) | Block at tool_call if asset exceeds this. | | mimeTypes | string[] | image/png, image/jpeg, image/gif, image/webp, image/bmp / audio/wav, audio/mpeg, audio/mp4, audio/ogg, audio/flac | Whitelist of MIME types to handle. |

Transcribe-specific fields

| Field | Type | Default | Notes | |---|---|---|---| | transcribe.provider | "minimax-cn" | "custom" | minimax-cn | Which back-end to call. | | transcribe.model | string | MiniMax-M3 | Model id passed to the transcribe endpoint. | | transcribe.prompt | string | (different per mode) | Instruction prompt sent alongside the asset. | | transcribe.maxOutputTokens | number | 1024 | Cap on the transcribe response. | | transcribe.timeoutMs | number | 60000 (images) / 120000 (audio) | HTTP timeout. | | transcribe.apiKeyEnv | string | MINIMAX_CN_API_KEY | Env var holding the API key. | | transcribe.endpoint | string | (auto) | Override URL for custom provider. | | transcribe.apiVersion | string | 2023-06-01 | Anthropic API version header. |

The config is re-read on every tool call, so toggling enabled / pipeline does not require /reload.

Tool registration

The extension registers a read_asset tool with one parameter:

{ path: string }

Internally, it routes by extension:

| Extension | Mode | Pipeline | Result | |---|---|---|---| | .png/.jpg/.jpeg/.gif/.webp/.bmp | images | inline | Tool result = [text confirmation, image base64] | | .wav/.mp3/.m4a/.ogg/.flac | audio | transcribe | Tool result = [text: "transcribed via X — description follows", text: description] | | audio with pipeline=inline | audio | inline | Tool result = error "audio inline is not supported in schemaVersion 3" | | (text file) | (none) | — | Tool returns error "use the regular read tool" |

The extension also injects a system prompt telling the LLM to use read_asset for binary files and read for text. If the LLM still tries to use the built-in read on a binary path, the call is blocked with a clear "use read_asset" message.

Verified with

• minimax-cn / MiniMax-M3 (Anthropic Messages API, multimodal for images)
• pi 0.84.1

Other multimodal models (Anthropic Claude 3.5+, OpenAI GPT-4o, Google Gemini 1.5+) should work the same way. Audio transcription relies on the transcribe endpoint accepting the asset — for minimax M3, image blocks are accepted (intermediate transcribe-via-image trick); true audio-input models would speak to the actual audio modality.

Limitations

• Audio transcribe is an image trick. The current transcribe implementation sends the audio as an image block to a model that accepts images but not audio. Vision models can ignore audio content. To support real audio input (gpt-4o-audio, Gemini Audio), the wire serializer needs to emit audio on the provider — not yet implemented in pi-ai 0.84.1.
• No image dedup. Reading the same image twice inlines it twice in the message history. /compact will compress old image blocks but not dedup.
• read_asset is a new tool. It does not appear in the LLM's knowledge of built-in tools. The system prompt injection covers most cases, but if the LLM ignores it, you'll need to explicitly say "use read_asset" in your prompt.
• No streaming for huge assets. A 5 MB image is ~5 MB base64 in memory. If you routinely hit the cap, raise it or use transcribe instead.

Security model

The extension's threat model is intentionally narrow. It only addresses what the plugin itself can do wrong; the user is responsible for everything else.

Plugin controls (designed in):

• Output sanitization. Tool result text is a fixed string ([inline-image-read] image X inlined / [inline-image-read] transcribed via Y). File contents enter the agent context only as raw image/audio blocks via the model serializer, never as plaintext. There is no path where user-controlled file bytes are echoed back as text.
• Pre-execution size cap. maxBytes is checked in the tool_call hook before the read tool runs, so an oversized file never enters the in-memory event payload.
• Type-segregated pipelines. pipeline: "inline" for audio refuses with a clear error rather than sending a wire format the upstream provider will reject (HTTP 400). Misconfigurations surface as readable errors, not silent failures.
• No eval, no shell, no network from the plugin itself. The plugin only reads files via node:fs and POSTs to one configured endpoint via fetch. No child_process, no require('http'), no dynamic code.

Out of scope (user's responsibility):

• Path safety. The plugin follows symlinks and reads any path the user passes to read_asset, including ..\..\..\..\Windows\..., ~/.secrets/..., \\?\GLOBALROOT\..., /proc/..., etc. Don't pass paths you don't want read. If you need a sandbox, run pi in one.
• Trusting the active model. Whatever the model does with the image (vision description, malicious interpretation, etc.) is the model's problem, not the plugin's. The plugin only delivers the bytes; it does not evaluate the response.
• Trusting the transcribe endpoint. The transcribe.endpoint / transcribe.apiKeyEnv settings let you point the audio pipeline at any URL. Whatever that endpoint does with your audio is between you and them. Defaults to minimax-cn/MiniMax-M3, which is the same provider pi uses for the main model — same trust boundary, same key.
• Memory pressure from many large reads. A batch of 10×5MB images fills 50MB of session memory. The plugin caps each file at maxBytes but does not cap total session memory; tune your thresholds if you work with large libraries.
• Trusting your settings.json. The user owns the file. Anyone who can write to ~/.pi/agent/settings.json can change transcribe.prompt to anything — including instructions that get sent to the transcribe model. That's the same threat model as any settings file.

License

MIT