HypGrep

Build a compact n-gram search index for a Parquet file using hyparquet and hyparquet-writer. Queries are case-insensitive substring matches — grep semantics over a precomputed index.

Why?

Enable efficient grep-style search on large Parquet datasets from any client without a server. Store your Parquet dataset on S3, generate a compact index file, and query it directly from a browser or other clients using HTTP range requests. The index tells you exactly which row blocks to fetch, so you only download the data you need.

Perfect for serverless architectures where you want to offer search capabilities without managing infrastructure.

CLI usage

Build an index:

npx hypgrep dataset.parquet [dataset.index.parquet]

Grep against the indexed file:

npx hypgrep search dataset.parquet 'serverless'          # literal substring
npx hypgrep search dataset.parquet '/eigen.+value/i'      # regex
npx hypgrep search dataset.parquet 'rhythm' --limit 5     # first N matches
npx hypgrep search dataset.parquet 'rhythm' -c            # count only
npx hypgrep search dataset.parquet 'rhythm' -i            # case-insensitive literal

To install as a system-wide CLI tool:

npm install -g hypgrep
hypgrep search dataset.parquet 'pattern'

Find rows in a parquet file in JavaScript

Use parquetFind to find rows containing the query as a substring while preserving natural row order (like Ctrl+F):

import { parquetFind } from 'hypgrep'

for await (const row of parquetFind({
  query: 'serverless',
  url: 'https://s3.hyperparam.app/hypgrep/wiki_en.parquet',
})) {
  console.log(row) // { title: '...', text: '...' }
}

The query matches as a contiguous substring (grep semantics): 'speed of light' matches rows containing that exact phrase, not rows where the words merely co-occur. Queries shorter than the indexed n-gram length (default 5) fall back to a full scan but still return correct results.

Regex queries

Pass a RegExp directly — mandatory literals are extracted from the pattern for index pruning, and regex.test runs against each row:

for await (const row of parquetFind({
  query: /eigen\w*value/i,
  url: '...',
})) ...

If the regex has no extractable literal (e.g. /./, /foo|bar/), the index can't prune and HypGrep does a full scan. The substring/regex filter still applies — results are correct, just unaccelerated.

If you want full control over the row predicate (e.g. a custom JS function), pass rowFilter. The string query is still used for index pruning while the callback decides which rows to keep:

for await (const row of parquetFind({
  query: 'eigen',
  rowFilter: row => myCustomCheck(row),
  url: '...',
})) ...

Ranked search

Use parquetSearch for Google-style ranked search: whitespace-separated words are ANDed (every word must appear), and results are ranked by total occurrence count:

import { parquetSearch } from 'hypgrep'

for await (const row of parquetSearch({
  query: 'serverless',
  url: 'https://s3.hyperparam.app/hypgrep/wiki_en.parquet',
})) {
  console.log(row) // most matches first
}

Create an index in JavaScript

import { asyncBufferFromFile } from 'hyparquet'
import { fileWriter } from 'hyparquet-writer'
import { createIndex } from 'hypgrep'

// Generate dataset.index.parquet from dataset.parquet
const sourceFile = await asyncBufferFromFile('dataset.parquet')
const indexFile = fileWriter('dataset.index.parquet')
await createIndex({ sourceFile, indexFile })

Local parquet files

To search against local parquet files, provide an asyncBufferFactory that loads the file from the local filesystem:

import { asyncBufferFromFile } from 'hyparquet'
import { parquetFind } from 'hypgrep'

// Loads parquet file from local filesystem
function asyncBufferFactory({ url }) {
  return asyncBufferFromFile(url)
}

for await (const row of parquetFind({
  query: 'serverless',
  url: 'dataset.parquet',
  asyncBufferFactory,
})) {
  console.log(row)
}