lexirank

Keyword knowledge retrieval — rank text items by lexical overlap. Zero dependencies, no vector DB.

You have a small knowledge base (services, FAQs, docs, products) and an incoming message. You want the few most relevant entries to drop into an LLM prompt — but you don't want to run embeddings, a vector database, or a network call to do it.

lexirank is that layer. It's a handful of pure functions that score items by keyword overlap and hand you back the top matches. No state, no I/O, no deps.

message ──▶ lexirank(query, items) ──▶ [ranked items] ──▶ your prompt
             (pure, in-process)

Install

npm install @zuperlana/lexirank

Node ≥ 18, ESM.

30-second example

import { selectRelevant } from '@zuperlana/lexirank';

const services = [
  { id: 1, text: 'HydraFacial', category: 'Facials',
    keywords: ['glow', 'skin', 'pores'], meta: { price: '$180' } },
  { id: 2, text: 'Balayage', category: 'Hair',
    keywords: ['color', 'highlights', 'blonde'], meta: { price: '$250' } },
];

const results = selectRelevant('my skin needs a glow', services);
// [ { item: { id: 1, text: 'HydraFacial', ... }, score: 4, matched: ['skin','glow'] } ]

const promptBlock = results
  .map(r => `${r.item.text} — ${r.item.meta.price}`)
  .join('\n');

That's the whole idea: data in as a parameter, ranked data out. How you format it for your prompt is up to you.

Try it live:

npm run demo

The data shape

Every entry is a KnowledgeItem. Only id and text are required.

interface KnowledgeItem {
  id: string | number;
  text: string;                     // primary field to match
  keywords?: string[];              // synonyms / tags — key for broad queries
  category?: string;                // optional, also matched
  meta?: Record<string, unknown>;   // price, answer, duration… never inspected
}

meta is yours. lexirank never looks inside it — it rides along so you have everything you need when you format the results.

API

selectRelevant(query, items, options?) → ScoredItem[]

Ranks items for a query. Selection order (first non-empty wins):

1. scored — items whose keyword overlap beats minScore
2. broad — if the query looks like "list everything", return the first N items
3. follow-up — if the query refers back ("how much is it?") and you passed history, return items mentioned earlier in the conversation
4. otherwise []

Fallback results (branches 2–3) carry score: 0 and matched: [] — you can branch on results[0]?.score to tell a real keyword hit from a fallback.

selectRelevant('what do you offer?', items, { broadLimit: 8 });
selectRelevant('how long does that take?', items, { history: previousLines });

scoreItem(query, item, options?) → ScoredItem

The scoring core. Pure: scores one item, returns { item, score, matched }. Useful when you want to rank/filter yourself.

createSelector(defaults?) → (query, items, options?) => ScoredItem[]

Bakes in defaults (weights, thresholds, match mode) once:

const select = createSelector({ limit: 3, weights: { keyword: 3 }, matchMode: 'word' });
select(message, services);

Options

| Option | Default | What it does | |---|---|---| | limit | 5 | Max results returned | | minScore | 1 | Exclusive threshold — items scoring ≤ this are dropped | | minWordLength | 3 | Ignore words shorter than this | | weights | { text: 2, keyword: 2, category: 2 } | Points per matched word, per field. 0 disables a field | | matchMode | 'substring' | 'substring' (fast, loose) or 'word' (word-boundary, precise) | | scorer | — | Custom scoring function (query, item) => number; replaces lexical scoring (the semantic term still applies) | | stopwords | none | Words to ignore while scoring. Pass your own or the exported DEFAULT_STOPWORDS | | dedupe | false | Count each distinct word once per field (a repeated word won't inflate the score) | | queryVector | — | Query embedding — turns on hybrid scoring (see below) | | semanticWeight | 4 | Points awarded for a perfect (cosine = 1) semantic match | | history | [] | Recent conversation lines (newest last) for follow-up recall | | broadMatch | built-in regex | true/false to force, or your own RegExp | | followupMatch | built-in regex | true/false to force, or your own RegExp | | broadLimit | = limit | How many items to return for a broad query |

Hybrid mode (optional semantic blend)

Keyword matching misses paraphrases that share no words ("complexion" ↔ "facial"). If you already have embeddings, lexirank can blend them in — while staying zero-dependency and synchronous: you compute the vectors with whatever model you like, lexirank just does the math.

// Attach a precomputed embedding to each item, pass the query's embedding in.
const items = [
  { id: 1, text: 'HydraFacial', vector: await embed('HydraFacial') },
  { id: 2, text: 'Haircut',     vector: await embed('Haircut') },
];

const results = selectRelevant('something for my complexion', items, {
  queryVector: await embed('something for my complexion'),
  semanticWeight: 5,   // how much a perfect semantic match is worth, in points
});

Final score = lexical points + semanticWeight × cosine(queryVector, item.vector). A strong semantic hit can clear minScore on its own, so items with no shared words still surface. Items without a vector (or a mismatched length) simply skip the semantic term — mixing vectorized and plain items is fine.

Stopwords & dedupe

For prose queries, common words ("the", "what", "for") can pad scores. Filter them out, and optionally stop repeated words from stacking:

import { selectRelevant, DEFAULT_STOPWORDS } from '@zuperlana/lexirank';

selectRelevant('what do you have for the skin', items, {
  stopwords: DEFAULT_STOPWORDS,  // or your own array / Set
  dedupe: true,                  // "skin skin skin" counts as one hit
});

DEFAULT_STOPWORDS is a small English list you opt into — nothing is filtered by default.

When to use it (and when not)

Use it when your knowledge base is small-to-medium, the vocabulary is predictable, and you want retrieval that's instant, free, and debuggable (the matched array tells you exactly why something ranked). Great for receptionist bots, FAQ pickers, command palettes, and prompt-context selection.

Reach for embeddings instead when you need semantic matching across large corpora or paraphrases that share no words ("cardiac" ↔ "heart"). lexirank matches words, not meaning — give it good keywords and it goes a long way, but it isn't a vector search.

License

MIT