Introduction

Commitspark is a set of tools to manage structured data with Git through a GraphQL API.

This package provides a keyword search adapter for the Commitspark GraphQL API. It is based on MiniSearch, ranks field values marked as searchable using the BM25+ ranking function, and runs entirely in memory without external services. Query terms also match terms they are a prefix of and, within limits, misspelled terms.

Installation

npm i @commitspark/search-adapter-minisearch

Usage

import { createAdapter as createSearchAdapter } from '@commitspark/search-adapter-minisearch'
import { createClient } from '@commitspark/graphql-api'

const gitAdapter = // ... any Commitspark Git adapter

const client = await createClient(gitAdapter, {
  searchAdapter: createSearchAdapter(),
})

See the Commitspark GraphQL API documentation on how to mark fields as searchable and how to query for search hits.

Options

| Option | Default | Description | |-----------------|------------|----------------------------------------------------------------------------------| | maxIndexes | 10 | Number of commits whose index is kept in memory | | analyzeWord | foldWord | Turns a single word into the terms to index and search for (see below) | | prefix | true | Whether query terms also match by prefix, e.g. launc matches launch | | fuzzy | 0.2 | Maximum edit distance for matching misspelled terms, or false to disable (1) | | snippetLength | 200 | Maximum snippet length in characters; longer values are excerpted around a match |

(1) Values below 1 are relative to the length of each query term, e.g. 0.2 allows one edit for query terms of 3 to 7 characters. Values of 1 or more are absolute.

Language-specific analysis

Values and queries are split into words consisting of letters and digits. By default, each word is lowercased and diacritics are removed, so that e.g. Café matches cafe. This works for any language but does not match different forms of a word, e.g. rockets does not match rocket (while rocket matches rockets by prefix).

To improve matching for the language of your content, provide your own analyzeWord function, e.g. based on a stemming library. The function receives a single word and returns the terms to use for it. Returning no terms drops the word (e.g. for stopwords), returning multiple terms allows e.g. splitting compound words:

import { createAdapter, foldWord } from '@commitspark/search-adapter-minisearch'

const stopwords = new Set(['a', 'an', 'the'])

const searchAdapter = createAdapter({
  analyzeWord: (word) =>
    foldWord(word)
      .filter((term) => !stopwords.has(term))
      .map((term) => stem(term)), // e.g. a stemming function for the language of your content
})

How it works

On the first search in a commit, the adapter retrieves all searchable field values of the commit and builds an in-memory index for it, which is reused for all further searches in the same commit. The indexes of the most recently used commits are kept, up to maxIndexes.

A search returns all values containing at least one of the query's terms, ranked by BM25+ score. The score is higher for values containing more of the query terms, terms that are rare across all values, and for shorter values. Exact matches score higher than prefix matches, which in turn score higher than matches of misspelled terms.

As indexes are held in memory, they are rebuilt after a restart of your application, e.g. on each cold start in serverless environments. As a rough guide, building an index for 100,000 values of a few dozen words each takes about a second and requires about 150 MB of memory. Searches take a few milliseconds, or up to a few hundred milliseconds for queries of very common terms that match most values.

Limitations

• Queries are not interpreted, i.e. there is no support for phrases, required terms or boolean operators.

License

The code in this repository is licensed under the permissive ISC license (see LICENSE).