@blahai/lens

BLAH Lens Bundle v1 — the stack-neutral interoperability format for model-internals analysis on evals.blah.dev, and the conformance validator for it.

Bundle format:    blah-jacobian-lens (version 1)
Runtime protocol: blah-lens-http/1
HF library tag:   blah-jlens

Nothing here is named after or coupled to any model framework, and nothing here executes model code. A bundle produced by PyTorch, JAX, MLX, Candle, Rust, Zig or a bespoke TypeScript engine is indistinguishable to the consumer.

Validate a bundle

npx @blahai/lens validate ./dist/blah-lens
npx @blahai/lens validate --hf your-org/model-name-jlens --revision <sha>
npx @blahai/lens validate-runtime --url http://localhost:8000 \
  --manifest ./dist/blah-lens/lens-manifest.json

Exit codes: 0 pass, 1 fail, 2 partial, 64 usage error. --json <file> writes the full report; --quiet prints only the verdict.

This is the same validator evals.blah.dev runs when a bundle is imported, so a bundle cannot pass here and be rejected there. If you would rather not install anything, the platform exposes it over HTTP:

curl -sX POST https://evals.blah.dev/api/v1/lens/validate \
  -H "Content-Type: application/json" \
  -d '{"lens_repo":"your-org/model-name-jlens","lens_revision":"<sha>"}'

Applying a transport

A Jacobian describes change, not absolute state, so the honest application is affine:

transported = target_mean + (h_s - source_mean_s) @ transpose(J_s)

Declare lens.centering.mode: "affine" with a target_mean_key and a source_mean_key per site; the validator then requires those tensors and checks their shapes. Uncentred application stays available and is warned about.

Declare which derivative you fitted in lens.estimator_kind:

| kind | meaning | |---|---| | same_position | d target[t] / d site[t] — what a token-aligned grid shows | | lagged | d target[t+lag] / d site[t] — influence on a later position | | future_integrated | summed over all later positions — downstream influence, not a prediction at t |

For same_position and lagged, also declare how the position-diagonal was isolated. This is not bookkeeping: a vector-Jacobian product with the cotangent injected at every target position at once returns sum over u >= t of the per-position derivatives, so the batched backward fits future_integrated while producing a matrix indistinguishable from a correct diagonal fit.

"diagonal_estimator": { "method": "isolated" }
"diagonal_estimator": { "method": "hutchinson", "probes": 32, "seed": 0 }

isolated is one backward per target position, sliced at the matching source position — exact, at T passes per direction. hutchinson carries Rademacher signs on the position index so cross-position terms cancel in expectation, giving an unbiased diagonal in one pass; the off-diagonal mass becomes variance falling as 1/sqrt(probes), which can flip the argmax of a top-1 readout, so the probe count is required and low-margin readouts should be treated as undetermined.

What it checks

The manifest schema encodes the mistakes that produce plausible-looking but wrong readouts, and rejects them:

• a transport whose output width is not the target width — the stored convention is J[output_dimension, input_dimension], so a silent transpose is caught
• a site claiming logit_lens_supported from an incompatible width, which would require inventing an implicit projection
• a mutable branch name where an immutable commit SHA belongs
• grid_compatible asserted alongside sites that are not token-aligned
• duplicate site IDs or orders, which would make grid rows ambiguous
• remote_http execution without a runtime URL, or oci without an image digest
• affine centering declared without the mean tensors it requires

Safetensors headers are read directly, without a tensor library, and over ranged requests where the reader supports them — validating a multi-gigabyte transport costs kilobytes. Pickle and other executable serializations are rejected.

Runtime conformance exercises what production actually calls, so a server cannot pass validation and then behave differently when the UI connects. It catches streams that never terminate, meta arriving after results, readouts attributed to undeclared sites, out-of-order top-k ranks, a runtime serving a different checkpoint than the bundle was fitted on, and a runtime that retokenizes a sequence it was asked to replay verbatim.

Library

import { validateBundle, directoryReader, LensManifest } from "@blahai/lens";

const report = await validateBundle(directoryReader("./dist/blah-lens"));
if (report.status !== "pass") console.error(report.checks.filter((c) => c.status === "fail"));

Subpath exports: @blahai/lens/manifest, /protocol, /safetensors, /validate, /runtime.

Producing a bundle

The author guide, a copy-paste prompt for a coding agent, and JSON Schemas for the manifest and protocol:

• https://evals.blah.dev/docs/lens
• https://evals.blah.dev/lens/agent-prompt.txt
• https://evals.blah.dev/lens/manifest.schema.json
• https://evals.blah.dev/lens/protocol.schema.json

Byte-level vocabularies

A byte token such as 0xC3 is not valid UTF-8 and has no faithful JSON string. Send its bytes alongside the display form; bytes_base64 is authoritative wherever present.

{ "id": 195, "text": "<0xC3>", "bytes_base64": "ww==" }

Scope

This is a readout instrument. It is not a sparse autoencoder, and a layer-to-readout Jacobian is not by itself evidence about global workspaces, broadcast or ignition — those need interventions and a control model. Describe what you publish accordingly.

MIT.