@volter/twin-turbopuffer

A local, stateful twin of the Turbopuffer API (v2 namespaces), built on @volter/world-core. The unmodified @turbopuffer/turbopuffer client writes documents into it and queries them back: filters, BM25 full-text ranking, attribute ordering, exact vector distance and counts are computed from the stored documents, deterministically.

The motivating consumer is Dub's partner search (apps/web/lib/api/partners/search/providers/ turbopuffer.ts): its sync jobs write upsert_rows with a declared schema and deletes, its search runs a multiQuery of BM25 branches with last_as_prefix, ContainsAllTokens and program / group / tag filters, its count runs aggregate_by: { total: ['Count'] }, and its maintenance script calls deleteAll.

Pointing a client at it

world-turbopuffer serve --port 8787 --root ./state
TURBOPUFFER_API_KEY=anything TURBOPUFFER_BASE_URL='http://127.0.0.1:8787/{region}' node app.js

In a World, volter-world init wires this for you: the service exports TURBOPUFFER_TWIN_URL and TURBOPUFFER_BASE_URL=${url}/{region}. The {region} placeholder is required: the SDK throws "region is set, but would be ignored" when a caller passes region (Dub passes aws-us-east-1) and the base URL has no placeholder. The twin accepts and ignores the leading region segment.

Grounded wire

Grounded in the installed @turbopuffer/[email protected] (src/ paths below are that package's).

| Fact | Value | Source | |---|---|---| | Default host | https://{region}.turbopuffer.com, e.g. aws-us-east-1.turbopuffer.com | src/client.ts:273 | | Region | region option or TURBOPUFFER_REGION; required when the URL has {region} | src/client.ts:255, 274-286 | | Base URL override | TURBOPUFFER_BASE_URL (or baseURL); {region} is substituted into it | src/client.ts:253, 281 | | Auth | Authorization: Bearer <apiKey> (TURBOPUFFER_API_KEY) | src/client.ts:254, 383 | | Request headers | Accept: application/json, Accept-Encoding: identity, User-Agent, X-Stainless-* | src/client.ts:858-864 | | Request body | JSON; gzip only with compression: true (default false) | src/client.ts:299, 812 | | Node transport | undici Agent.request, selected by the #fetch import map for the node condition | package.json imports, src/internal/custom/fetch-node.ts | | Write | POST /v2/namespaces/{ns} | src/resources/namespaces.ts:174 | | deleteAll | DELETE /v2/namespaces/{ns} → { status: 'OK' } | namespaces.ts:46, 1100 | | Query | POST /v2/namespaces/{ns}/query | namespaces.ts:104 | | multiQuery | POST /v2/namespaces/{ns}/query?stainless_overload=multiQuery with { queries } | namespaces.ts:90 | | Write response | { status: 'OK', message, rows_affected, rows_upserted?, rows_patched?, rows_deleted?, *_ids?, billing } | namespaces.ts:1231-1294 | | Query response | { rows?, aggregations?, billing, performance }; a row is { id, $dist?, ...attributes } | namespaces.ts:850-868, 1159-1175 | | FTS defaults | tokenizer word_v4, k1 1.2, b 0.75, case-insensitive, stopwords kept (remove_stopwords false since January 2026), no stemming, max_token_length 39 | namespaces.ts:474-526 | | Filterable default | full-text attributes are not filterable unless filterable: true | namespaces.ts:249-253 | | Errors | status → BadRequestError 400, AuthenticationError 401, NotFoundError 404, … | src/core/error.ts |

Under Node the SDK sends through undici's Agent.request, which neither the injector's http/https/fetch patches nor HTTPS_PROXY reach, so TURBOPUFFER_BASE_URL is the interception. The descriptor also claims the regional hosts (^(aws|gcp|azure)-….turbopuffer.com$) for runtimes that use the fetch build (Bun, edge).

Coverage

Modelled:

• write: upsert_rows, upsert_columns, patch_rows, patch_columns, deletes, delete_by_filter, patch_by_filter, schema, distance_metric, return_affected_ids, disable_backpressure (accepted; the twin has no backpressure). One write request is one kernel action. Extrapolated, not in the SDK: the twin applies it in the order filtered deletes → filtered patches → upserts → patches → deletes. Upserts replace the whole document; patches merge and never create; a null value removes the attribute. Attribute types are declared or inferred on first write (string, int, float, bool, []string, …) and a mismatched value or type change is refused with 400.
• query: filters (Eq, NotEq, In, NotIn, Contains, NotContains, ContainsAny, NotContainsAny, Lt/Lte/Gt/Gte, AnyLt/AnyLte/AnyGt/AnyGte, Glob/NotGlob/IGlob/NotIGlob, ContainsAllTokens, ContainsAnyToken with last_as_prefix, And/Or/Not); rank_by BM25 (with last_as_prefix, Sum, Max, Product), attribute order ([attr, asc|desc] and lists of them), and [vector, ANN|kNN, q] with exact cosine_distance / euclidean_squared; top_k / limit (default 10); include_attributes / exclude_attributes; aggregate_by Count, Count(attr), Sum(attr); consistency (the twin is always strongly consistent); multiQuery.
• namespaces: deleteAll. A query on a namespace that does not exist answers 404. Listing, metadata, the schema endpoints and the cache-warm hint are called by no application here (journeys/demand.json) and answer the gap.
• auth: a missing or empty Bearer key is 401; a key the dashboard (or the World's credential door, which issues the application's TURBOPUFFER_API_KEY) never made, or one it expired, is 401.
• no state system (the gap): a World's writes are not performed against turbopuffer, and a real account's namespaces are not observed.

Semantics a caller depends on: an attribute a document does not carry reads as null, so NotIn, NotEq and NotContainsAny match it (Dub omits groupId/country for exactly this). A comparison filter on a non-filterable attribute is refused with 400; the token filters require full_text_search on the attribute.

Deterministic, not byte-equal

Scores and tokens are deterministic but are not Turbopuffer's exact numbers. word_v2 is modelled as maximal runs of Unicode letters, digits, marks and _ (so a URL splits into its components, as Dub measured); English stopwords are the 33-word Lucene list; BM25 is textbook Okapi with the Lucene IDF ln(1 + (N − n + 0.5)/(n + 0.5)) over the namespace's documents carrying the attribute; a last_as_prefix final token contributes a constant 1 per matching document (Dub observed that a single-token prefix query scores every match exactly 1). Ties order by id ascending. Error message strings and the { status: 'error', error } body are the twin's own reading (this build had no account to probe); the SDK surfaces that body verbatim as APIError.error, and the statuses map to the SDK's error classes. billing and performance are deterministic (logical bytes from the stored JSON, zero timings, a "hot" cache).

Also the twin's own choices, not grounded in the SDK: namespace names [A-Za-z0-9-_.]{1,128}; page_size at most 1000; deletes counts every requested id in rows_deleted; an undeclared integer attribute is inferred as int; an attribute a row lacks projects as null when named in include_attributes; a token filter whose every token is dropped (stopwords) matches nothing; id comparisons order numbers before strings.

Planned (todo)

Refused with a 400 or 404 naming the gap, never answered with a fabricated success: conditional writes (upsert_condition / patch_condition / delete_condition), copy_from_namespace, branch_from_namespace and their asynchronous operations, encryption, sharding, write backpressure, metadata PATCH (pinning), explain_query, recall, group_by, compute_attributes, aggregate_by together with rank_by, limit.per, rerank_by, rank expressions beyond BM25/Sum/Max/Product, highlights, Regex / Fuzzy / ContainsTokenSequence filters, tokenizers other than word_v2 and pre_tokenized_array (including the default word_v4, so full_text_search: true is refused), stemming, non-English languages, byte-equal BM25 scores, base64 vectors, named / multi / sparse vectors, server-side embedding, the v1 namespace API older SDKs call, and the vendor's literal error strings.

The evidence recorded when the pack was built was the real @turbopuffer/[email protected] client, constructed as Dub constructs it, performing Dub's write → multiQuery → count → delete → deleteAll sequence against a served twin through TURBOPUFFER_BASE_URL.

No UI mirror

Turbopuffer is used from code: an application writes and queries namespaces through the SDK. The turbopuffer.com dashboard manages API keys, billing and namespace inspection; the work does not happen there.

Layout

| File | Role | |---|---| | src/fetch.ts | the region segment, the API key gate, the derived dispatch | | src/server.ts | the discovery door, the dashboard, the server | | src/semantics/namespaces.ts | the handlers, one per operation | | src/semantics/shared.ts | the namespaces from the tree, the error body, a write landed | | src/engine/namespace.ts | namespaces and documents; the pure write and query | | src/engine/filter.ts | the filter language | | src/engine/text.ts, src/engine/stem.ts | tokenizer, English stemming and BM25 | | src/screens/api-keys.tsx | the dashboard's sign-in and API keys |

A World's writes are not performed against turbopuffer and its namespaces are not observed from it (no state system: the gap).