HypStore

HypStore is a serverless lakehouse for JavaScript: a store of named Apache Iceberg tables in object storage you control, where every table can be queried three ways: SQL, full-text grep, and vector similarity. Writes and queries run client-side (browser or node.js) using HTTP range requests. No server, no always-on infrastructure.

Part of HypStack, an open-source stack for AI observability.

HypStore ties together the HypStack libraries into one coherent store:

• icebird: reads and writes the Iceberg tables (append, delete, compact, time travel)
• squirreling: streaming SQL across tables
• hypgrep: full-text search via n-gram indexes
• hypvector: similarity search over embedding vectors

Append-heavy, text-rich data is the sweet spot (AI logs, traces, and agent transcripts most of all), but the interface is a general table store.

Usage

createStore returns a plain immutable config object, and every operation is a standalone function that takes the store as an argument.

import { createStore } from 'hypstore'
import { s3SignedResolver } from 'icebird'

const store = createStore({
  warehouseUrl: 's3://my-bucket/warehouse',
  resolver: s3SignedResolver({ accessKeyId, secretAccessKey, region: 'us-east-1' }),
  // optional: bring your own embedding function for vector search
  embedder: async texts => embed(texts), // returns Float32Array[]
})

A store holds many tables. Each table declares its schema and, optionally, which columns get text and vector indexes:

import { createTable } from 'hypstore'

await createTable({
  store,
  table: 'articles',
  schema: {
    id: 'string',
    title: 'string',
    body: 'string',
    published: 'timestamp',
  },
  index: {
    text: ['title', 'body'],  // hypgrep n-gram index
    vector: ['body'],         // hypvector embeddings (requires embedder)
  },
})

Append

import { append } from 'hypstore'

await append({
  store,
  table: 'articles',
  records: [
    { id: 'a1', title: 'On Serverless Search', body: '...', published: new Date() },
  ],
})

append commits rows via icebergAppend and incrementally maintains the declared indexes alongside the data files.

SQL

import { collect, sql } from 'hypstore'

const results = await sql({
  store,
  query: 'SELECT title, published FROM articles WHERE published > CURRENT_DATE - INTERVAL \'7\' DAY ORDER BY published DESC',
})
const rows = await collect(results)

Rows stream lazily via squirreling, so LIMIT queries over huge tables only fetch the bytes they need. Queries can join across any tables in the store.

Grep

import { grep } from 'hypstore'

for await (const row of grep({ store, table: 'articles', query: 'range request' })) {
  console.log(row.title)
}

Grep semantics: contiguous substring or regex match on indexed text columns, accelerated by the hypgrep index to fetch only the row blocks that can match. Data files without an index (written before indexing was declared, or by another tool) fall back to a full scan, so results are always correct.

Search

import { search } from 'hypstore'

const hits = await search({
  store,
  table: 'articles',
  query: 'how do I query parquet without a server',
  topK: 10,
})

The query is embedded with the store's embedder and matched against the hypvector index. Pass a Float32Array directly to skip embedding.

Add a SQL where predicate to rank only among matching rows. Search over-fetches candidates and widens the pool automatically until it fills topK:

const hits = await search({
  store,
  table: 'articles',
  query: 'cost of always-on infrastructure',
  topK: 10,
  where: "published > CURRENT_DATE - INTERVAL '1' DAY",
})

Maintenance

import { compact } from 'hypstore'

await compact({ store, table: 'articles' }) // icebergRewrite + rebuild indexes over the compacted files

Architecture

An Iceberg table is a set of Parquet data files plus snapshot metadata. hypgrep and hypvector index individual Parquet files, so HypStore maintains one index file per data file:

s3://bucket/warehouse/articles/
  metadata/            Iceberg snapshots, manifests
  data/                Parquet data files (written by icebird)
  index/
    <file>.grep.parquet    n-gram index per data file (hypgrep)
    <file>.vec.parquet     embedding vectors per data file (hypvector)

Queries fan out across the live data files of the current snapshot, apply Iceberg deletes (position deletes, deletion vectors, and equality deletes), and merge results: top-K merge for vector search, ordered merge for grep. Compaction rewrites data files and their indexes together and deletes orphaned index files, so indexes never go stale.

AI logs

The primary use case HypStore is designed around: capture LLM calls, traces, and agent transcripts, then debug with grep, analyze with SQL, and triage with semantic search. A ready-made schema preset ships with the package:

import { aiLogSchema, createTable } from 'hypstore'

await createTable({ store, table: 'logs', ...aiLogSchema })

| Column | Type | Notes | |---|---|---| | id | string | unique record id | | timestamp | timestamp | event time (partition column) | | model | string | model id | | role | string | system / user / assistant / tool | | content | string | message text (text + vector indexed) | | metadata | string | JSON blob for arbitrary attributes |

See EXAMPLES.md for full usage patterns, including the AI-log workflows.

Status

The interface described above is implemented and tested: table creation, indexed appends, SQL (including joins, async UDFs, and asOf time travel), grep, vector search with where filtering, Iceberg delete handling, compaction with index rebuilds, and snapshot expiry. The test suite covers unit tests plus end-to-end integration against real filesystem and HTTP range-request storage. Not yet published to npm.

References

• icebird
• squirreling
• hypgrep
• hypvector
• Iceberg spec