cry-search

A fast, memory-efficient search library for large datasets with support for tokenized matching, linked collections, and per-field match modes.

Features

• Fast search - Pre-built metadata enables instant queries on large datasets
• Memory efficient - Numeric token storage (Uint32Array) keeps metadata compact
• Updatable - Add, update, or remove items without rebuilding metadata
• Augmented metadata - Add local/temporary searchable data without modifying global metadata
• Flexible matching - Per-field match modes and query prefixes for prefix, suffix, anywhere, exact, and negation
• Search prefixes - use -word, --word, or ~word to exclude, ..word for suffix, =word for exact match
• Automatic normalization - Handles diacritics, case, dates, and number formatting
• Ranked search - searchAndRank returns results scored by positional relevance (token order, contiguity, perfect-prefix, length)
• Universal search - Search and update across linked collections (e.g., customers with their pets)

Table of Contents

• cry-search
  • Features
  • Table of Contents
  • Installation
  • Examples
    • Single Table Search
    • SearchUniverse with Linked Collections
    • Updating Existing Data
    • Augmented Metadata - Local Search Context
    • Ranked Search with Positional Scoring
  • Architecture
    • Text Processing Pipeline
    • Metadata Building
    • Searching
    • Memory Optimization
  • Query Prefixes
  • Specification
  • License

Installation

npm install cry-search

Examples

Single Table Search

import {
  createSearchArrayMetadata,
  searchInDataReturnObjects,
  arrayToSearchableData,
  type SearchableObject,
} from 'cry-search';

interface Product extends SearchableObject {
  _id: string;
  name: string;
  description: string;
  sku: string;
}

const products: Product[] = [
  { _id: '1', name: 'iPhone 15 Pro', description: 'Latest Apple smartphone', sku: 'APL-IP15P' },
  { _id: '2', name: 'Samsung Galaxy S24', description: 'Android flagship phone', sku: 'SAM-GS24' },
  { _id: '3', name: 'MacBook Pro 16"', description: 'Apple laptop for professionals', sku: 'APL-MBP16' },
];

// Convert array to searchable data (Map<id, item>)
const data = arrayToSearchableData(products);

// Build search metadata (do this once, reuse for all searches)
const metadata = createSearchArrayMetadata(data, {
  searchInFields: ['name', 'description', 'sku'],
});

// Search
const results = searchInDataReturnObjects('apple', metadata, data);
// Returns: Map with products 1 and 3

const results2 = searchInDataReturnObjects('samsung galaxy', metadata, data);
// Returns: Map with product 2

// Search is diacritics and case insensitive
const results3 = searchInDataReturnObjects('IPHONE', metadata, data);
// Returns: Map with product 1

SearchUniverse with Linked Collections

For related data (like customers and their pets), use SearchUniverse to search across collections:

import { SearchUniverse, type SearchableObject } from 'cry-search';

interface Stranka extends SearchableObject {
  _id: string;
  name: string;
  address: string;
}

interface Pacient extends SearchableObject {
  _id: string;
  stranka_id: string;  // Foreign key to Stranka
  name: string;
  species: string;
}

// Create universe and register collections
const universe = new SearchUniverse();

universe.registerCollection<Stranka>('stranke', {
  spec: { searchInFields: ['name', 'address'] },
});

universe.registerCollection<Pacient>('pacienti', {
  spec: { dontSearchInFields: ['stranka_id'] },
  linkedTo: {
    collectionName: 'stranke',
    foreignKeyGetter: (p) => p.stranka_id,
  },
});

// Load data
universe.loadCollection('stranke', [
  { _id: 's1', name: 'Krajnik', address: 'Ljubljana' },
  { _id: 's2', name: 'Novak', address: 'Maribor' },
]);

universe.loadCollection('pacienti', [
  { _id: 'p1', stranka_id: 's1', name: 'Angie', species: 'cat' },
  { _id: 'p2', stranka_id: 's1', name: 'Rex', species: 'dog' },
  { _id: 'p3', stranka_id: 's2', name: 'Bella', species: 'cat' },
]);

// Search across all collections
const results = universe.searchUniversally('krajnik angie');
// Returns:
// {
//   stranke: [{ _id: 's1', name: 'Krajnik', ... }],
//   pacienti: [{
//     primary: { _id: 's1', name: 'Krajnik', ... },
//     linked: [{ _id: 'p1', name: 'Angie', ... }],
//     matchedIn: 'both'
//   }]
// }

// Search only in linked items
const results2 = universe.searchUniversally('rex');
// Returns pacienti results with matchedIn: 'linked'

// Search only in primary - returns all linked items
const results3 = universe.searchUniversally('novak');
// Returns Novak stranka with Bella in linked array

Updating Existing Data

When data changes, update the metadata to keep search in sync:

import {
  createSearchArrayMetadata,
  updateSearchMetadata,
  updateSearchMetadataBatch,
  arrayToSearchableData,
} from 'cry-search';

// Initial setup
const data = arrayToSearchableData(products);
const metadata = createSearchArrayMetadata(data);

// Update a single item
const updatedProduct = { ...products[0], name: 'iPhone 16 Pro' };
data.set(updatedProduct._id, updatedProduct);
updateSearchMetadata(metadata, updatedProduct._id, updatedProduct);

// Update multiple items at once
const updates = [
  { _id: '1', name: 'iPhone 16 Pro Max', description: 'Newest Apple phone', sku: 'APL-IP16PM' },
  { _id: '4', name: 'iPad Pro', description: 'Apple tablet', sku: 'APL-IPAD' },
];
for (const item of updates) {
  data.set(item._id, item);
}
updateSearchMetadataBatch(metadata, updates);

// Mark item as deleted (removes from search but keeps in data)
const deletedProduct = { ...products[1], _deleted: new Date() };
data.set(deletedProduct._id, deletedProduct);
updateSearchMetadata(metadata, deletedProduct._id, deletedProduct);

// With SearchUniverse - handles linked indexes automatically
universe.updateSearchMetadata('pacienti', 'p1', {
  _id: 'p1',
  stranka_id: 's2',  // Changed owner from s1 to s2
  name: 'Angie',
  species: 'cat',
});

Augmented Metadata - Local Search Context

Sometimes you need to search items combined with page-specific data (like match status, local flags, computed properties) without modifying the global metadata. AugmentedMetadata wraps existing metadata and allows adding temporary searchable data.

import {
  createSearchArrayMetadata,
  searchInDataReturnIds,
  AugmentedMetadata,
  arrayToSearchableData,
  type SearchableObject,
} from 'cry-search';

interface Medicine extends SearchableObject {
  _id: string;
  name: string;
  manufacturer: string;
}

// Global data - shared across application
const medicines: Medicine[] = [
  { _id: '1', name: 'Aspirin', manufacturer: 'Bayer' },
  { _id: '2', name: 'Paracetamol', manufacturer: 'Krka' },
  { _id: '3', name: 'Ibuprofen', manufacturer: 'Lek' },
];

const data = arrayToSearchableData(medicines);
const globalMetadata = createSearchArrayMetadata(data);

// Page-specific: matching medicines with external registry
// Create augmented metadata with local match status
const augmented = new AugmentedMetadata(
  globalMetadata,
  { searchInFields: ['status', 'notes'] }  // Control which fields to index
);

// User matches medicines with external registry
augmented.updateAugmented('1', { status: 'matched', notes: 'verified' });
augmented.updateAugmented('2', { status: 'unmatched', notes: '' });
augmented.updateAugmented('3', { status: 'matching in progress', notes: 'awaiting approval' });

// Search by base data + augmented status
const matchedAspirin = searchInDataReturnIds('aspirin matched', augmented);
// Returns: ['1']

const unmatchedItems = searchInDataReturnIds('unmatched', augmented);
// Returns: ['2']

const inProgress = searchInDataReturnIds('progress', augmented);
// Returns: ['3']

// Search with negation
const notMatched = searchInDataReturnIds('bayer -matched', augmented);
// Returns: [] (Bayer Aspirin is matched)

// Batch update for efficiency
augmented.updateAugmentedBatch(new Map([
  ['1', { status: 'matched', notes: 'final' }],
  ['2', { status: 'matched', notes: 'approved' }],
  ['3', undefined], // Remove augmentation
]));

// Clear specific item
augmented.clearAugmented('1');  // Reverts to global metadata only

// Clear all augmentations
augmented.clearAllAugmented();  // Reset to global state

// AugmentedMetadata is memory efficient:
// - Items without augmentation reuse original Uint32Array (no memory duplication)
// - Items with augmentation get merged sorted token array (base + augmented)
// - Global metadata remains unchanged

Use cases:

• Medicine matching with external registries (matched/unmatched status)
• Products with local pricing or availability flags
• Documents with review status or approval state
• Items with computed scores or temporary categories

Ranked Search with Positional Scoring

Since 2.0.0 all search functions return ranked results by default. searchInData* and findInLinkedArrays automatically score candidates and sort by relevance DESC. To activate positional scoring, declare spec.keyFields — those fields are tokenized in source order at build time and stored in metadata.unsorted for fast query-time scoring.

searchAndRank is a thin wrapper that additionally supports:

• exactKeysFn — +100 000 boost when raw query equals a designated field value (e.g. barcode)
• primaryTextFn — fallback when keyFields not declared, plus locale-aware tiebreaker sorting

import {
  arrayToSearchableData,
  createSearchArrayMetadata,
  searchAndRank,
  type SearchableObject,
} from "cry-search";

interface Product extends SearchableObject {
  _id: string;
  name: string;
  barcodes?: string[];
}

const products: Product[] = [
  { _id: "1", name: "POTROŠNI MATERIAL" },
  { _id: "2", name: "POTROŠNI MATERIAL-SUCRALAN 1ML" },
  { _id: "3", name: "KONTROLNI PREGLED" },
  { _id: "4", name: "DERMATOLOŠKI KONTROLNI PREGLED" },
];

const data = arrayToSearchableData(products);
// Declare keyFields so positional ranking has pre-stored ordered tokens (faster).
const meta = createSearchArrayMetadata(data, {
  keyFields: ['name'],
  firstTokensN: 8,
});

// Returns items sorted by relevance, NOT discovery order:
const results = searchAndRank("pot mat", data, meta, {
  primaryTextFn: (p) => p.name,
  exactKeysFn: (p) => p.barcodes,  // optional: exact-key boost (e.g. barcode)
  limit: 10,
});

// results[0].item.name === "POTROŠNI MATERIAL"        (perfect prefix → +80)
// results[1].item.name === "POTROŠNI MATERIAL-SUCRALAN 1ML"  (longer naziv)

// Each result: { id: Id, item: T, score: number }

Performance optimization via keyFields: When the spec declares keyFields, the first N tokens (firstTokensN, default 8) of those fields are stored in metadata.unsorted at build time. searchAndRank reads those at query time instead of re-tokenizing — typically 10-50× faster scoring. Without keyFields, searchAndRank falls back to tokenizing primaryTextFn(item) per candidate at query time (same correctness, slower).

Scoring formula (scoreOne is exported as a primitive — useful for composing your own scorer):

| Component | Value | When | |---|---|---| | Exact-key match | +100 000 | query === any(exactKeys) (e.g. barcode equality) | | Base | +50 | Any in-order subsequence-prefix match of query tokens in itemTokens | | All tokens matched | +100 | Every query token had a target | | First-word match | +50 | First match at position 0 | | Perfect prefix | +80 | Matches at positions [0,1,…,k-1] (contiguous from start) | | Density | +30 × (matched / span) | Rewards contiguous matches over gapped | | Length penalty | -0.1 × Σ itemToken.length | Weak tiebreaker — shorter wins | | Secondary-field-only | 1 | Item matched (via retrieval) but no query token appears in itemTokens |

Lower-level scoring primitives for composing custom rankers:

| Function | Use case | Order-sensitive? | |---|---|---| | scoreOne(query, qTokens, itemTokens, exactKeys?) | Full-featured (perfect prefix, density, length penalty, exact-key boost) — default used by searchInData* and searchAndRank | yes | | scoreOrderAndPresence(qTokens, itemTokens, weights?) | Simpler additive bonus: ordered match (+30) + presence (+10) per token. Tunable via weights | yes | | scoreBag(qTokens, itemTokens, perTokenWeight?) | Order-agnostic: each query token present in itemTokens contributes equally. Use for person names, tags, pet breeds where word order is conventional but not semantic — scoreBag(["janez","novak"], item) === scoreBag(["novak","janez"], item) | no | | orderedTokenize(input) | Tokenize without alphabetic sort (required for any positional scoring). Output is NOT compatible with searchInData* retrieval (which requires sorted lists from prepareStringForSearch) | n/a |

import { scoreBag, scoreOne, scoreOrderAndPresence, orderedTokenize, tokens, findTokenId } from 'cry-search';

// Combine signals per field in a custom ranker:
const qTokens = orderedTokenize(query);

// From metadata.unsorted (numeric IDs → strings via global tokens):
const unsortedIds = meta.unsorted.get(itemId);
const itemTokens = unsortedIds ? Array.from(unsortedIds, (id) => tokens[id]!) : [];

const baseScore = scoreOne(query, qTokens, itemTokens, item.barcodes);
const nameBoost = scoreBag(qTokens, nameTokens);                 // order-agnostic for "Janez Novak"
const titleBoost = scoreOrderAndPresence(qTokens, titleTokens);  // order-sensitive for "RTG slika"
const total = baseScore + nameBoost + titleBoost;

Architecture

cry-search uses a two-phase approach: build metadata once, then search instantly.

Text Processing Pipeline

Both metadata building and search queries go through the same pipeline:

Input: "Čokolada 27.12.2025 SI123"
   ↓
1. Normalize dates     → "Čokolada 20251227 SI123"
2. Preprocess          → "Čokolada 20251227 SI 123"  (split letter-digit)
3. Sanitize            → "cokolada 20251227 si 123"  (lowercase, remove diacritics)
4. Tokenize            → ["cokolada", "20251227", "si", "123"]
5. Sort                → ["123", "20251227", "cokolada", "si"]
6. Numeric IDs         → Uint32Array [42, 891, 156, 7]  (indices into global registry)

Metadata Building

Tokens are stored as numeric IDs in a global registry. Each item's tokens are stored in a sorted Uint32Array for memory efficiency and fast binary search.

Searching

Query tokens are matched against metadata using binary search. All query tokens must match for an item to be returned. Match modes (start, end, startEnd, anywhere, whole) control how tokens are compared.

Memory Optimization

Tokens are stored as Uint32Array indices into a global tokens: string[] registry. Each metadata entry is a compact sorted Uint32Array per item — tokens themselves are interned once globally.

Query Prefixes

Override match behavior per token at search time using prefixes:

| Prefix | Mode | Description | |--------|------|-------------| | word.. | start | Match tokens starting with "word" | | ..word | end | Match tokens ending with "word" | | =word | whole | Exact match only | | ?word | anywhere | Match "word" anywhere in token | | -word | negation | Exclude results (only when followed by letter) | | --word | negation | Exclude results (works for words and numbers) | | ~word | negation | Exclude results (works for words and numbers) |

Note: -5 is NOT negation — the leading dash is stripped, so -5 searches for 5. Use --5 or ~5 to negate numbers.

Detecting negation: Linked search results include a hasNegation: boolean flag indicating if the query contained negation tokens.

Alternative prefixes (for programmatic use): <word (start), >word (end), +word (startEnd), *word (anywhere), !word (whole)

// Find "jana" but only exact match on "usenik"
searchInDataReturnObjects('jana =usenik', metadata, data);
// Matches "Jana Usenik" but not "Jana Useniker"

// Find "apple" but exclude results containing "iphone"
searchInDataReturnObjects('apple -iphone', metadata, data);
// Matches MacBook Pro but not iPhone

// Find products with SKU starting with "APL"
searchInDataReturnObjects('APL..', metadata, data);
// Matches APL-IP15P, APL-MBP16, etc.

// Find exact barcode code
searchInDataReturnObjects('=9001', metadata, data);
// Matches "9001" but NOT "035585249001" or "9001917"

Number Matching

Match mode for numbers is determined by the target token's length (not the query):

• Short numbers (≤6 digits): start (left) match
• Long numbers (>6 digits): allow start or end match (for barcodes)

// Query "12" against code "1234" (4 digits) → MATCH (starts with 12)
// Query "60731" against barcode "3838989760731" (13 digits) → MATCH (ends with 60731)
// Query "34" against code "1234" (4 digits) → NO MATCH (not at start)
// Use "=12" for exact match on short numbers

Hyphen Splitting

Tokens are split at letter-dash-digit, digit-dash-letter and letter-dash-letter boundaries during preprocessing. Digit-dash-digit ranges (e.g., 12-25) are preserved.

// "ABC-00123"      → tokens "abc" and "123"
// "Ioana-Gabriela" → tokens "ioana" and "gabriela"
// "COVID-19"       → tokens "covid" and "19"
// "12-25"          → token "12-25" (range preserved)

This enables matching by either part:

// Query "123" against "ABC-00123" → MATCH (token "123" exists)
// Query "-123" against "ABC-00123" → MATCH (leading dash stripped → "123")
// Query "-0123" against "ABC-00123" → MATCH (leading zeros + dash stripped → "123")
// Query "abc 123" against "ABC-00123" → MATCH (both tokens match)
// Query "gabriela" against "Ioana-Gabriela" → MATCH (split into "ioana" and "gabriela")

Priority: query prefix > field match mode > global SearchOpts default.

Specification

See CLAUDE.md for the complete technical specification including:

• Input processing pipeline (date normalization, preprocessing, sanitization)
• Tokenization rules
• Matching behavior
• Linked search semantics
• Type definitions

License

See LICENSE.md for license terms.