@rantingjs/search

Full-text search on models. A searchable model is kept in its search index as it is saved, deleted and restored, and searched with Post.search('...'): through the database itself, or a search service (Meilisearch, Typesense, Algolia, Turbopuffer).

bunx ranting install @rantingjs/search

A searchable model

import { Model } from '@rantingjs/framework/db';
import { Searchable } from '@rantingjs/search';

export class Post extends Searchable(Model) {
  static table = 'posts';

  // What is indexed (the model's JSON by default).
  toSearchableArray() {
    return { id: this.id, title: this.title, body: this.body };
  }

  // Only published posts are in the index; a post that is unpublished leaves it.
  shouldBeSearchable() {
    return this.published;
  }
}

Searchable goes around the other mixins: class Post extends Searchable(SoftDeletes(Model)). Other hooks:

• searchIndexShouldBeUpdated(): whether a save updates the index (return this.$wasRecentlyCreated || this.wasChanged('title')).
• makeSearchableUsing(models): changes the models before they are indexed (return models.load('author')); makeAllSearchableUsing(query) the query an import reads them with (return query.with('author')).
• searchableAs(): the index name (the search.prefix and the table); indexableAs() the one written to.
• searchableUsing(): the engine of this model (return Search.engine('meilisearch')).
• getSearchKey() / getSearchKeyName(): the key the index uses (the primary key).

Searching

const posts = await Post.search('star trek').get();
const page = await Post.search('star trek').where('user_id', 1).paginate(15); // returned from a controller: data, total, links

| Method | | |---|---| | where(field, value), where(field, '>', value) | A filter every result matches (=, !=, <, >, <=, >=) | | whereIn(field, values), whereNotIn(field, values) | | | orderBy(column, 'desc'), orderByDesc, latest(), oldest() | | | take(n) | At most n results | | within(index) | Another index (a sorted replica, for example) | | withTrashed(), onlyTrashed() | Soft-deleted models too, or only them (with search.softDelete) | | withQuery((query) => query.with('author')) | Changes the query the found models are read with | | withOptions({ ... }) | The engine's own search parameters | | withRawResults((raw) => ...) | Sees (or replaces) the engine's raw results | | get(), first(), keys(), raw(), cursor() | Models, the first one, their keys, the engine's answer, models read lazily | | paginate(perPage, pageName, page), simplePaginate(...) | Pages; the query is kept in the links | | paginateRaw(...), simplePaginateRaw(...) | Pages of the engine's answer | | when(), unless(), tap() | |

A callback gets the engine's request (or, with the database and collection engines, the query) and runs the search itself: Post.search('star', (query) => query.where('featured', true)).

Keeping the index in sync

Saving, deleting, force deleting and restoring a searchable model update its index. By hand:

await Post.where('price', '>', 100).searchable();   // a query, in chunks (the ones that shouldBeSearchable)
await Post.where('archived', true).unsearchable();
await posts.searchable();                           // a collection
await post.searchable();                            // one model
await Post.makeAllSearchable();                     // every model (search:import)
await Post.removeAllFromSearch();                   // empties the index (search:flush)
await Post.withoutSyncingToSearch(() => Post.create({ ... })); // this work only; other requests still sync
Post.disableSearchSyncing(); Post.enableSearchSyncing();

ModelsImported and ModelsFlushed are dispatched per chunk of a query's searchable() / unsearchable().

With search.queue (SEARCH_QUEUE=true, or { connection, queue }) the index is updated by queued jobs: MakeSearchable (the class name and keys: the job reads the models again) and RemoveFromSearch (only the keys, so it works after the row is gone). Search.makeSearchableUsing(MakeSearchableUniquely) and Search.removeFromSearchUsing(RemoveFromSearchUniquely) skip copies already waiting. search.jobs sets their tries, backoff and exceptions. With search.afterCommit, a change made inside a transaction syncs once it commits, never on a rollback.

With search.softDelete, a soft-deleted model stays in the index marked __soft_deleted, for withTrashed() and onlyTrashed().

Engines

SEARCH_DRIVER chooses the engine.

| Engine | | |---|---| | collection (default) | Reads the rows that pass the filters and keeps those whose searchable fields contain the query (any case). Works on every database; for small tables, prototypes and tests. | | database | SQL: every searchable field with LIKE %query% (ilike on Postgres); static searchUsingPrefix = ['email'] fields from their start; static searchUsingFullText = ['body'] (or { columns, options: { language, mode } }) with the database's full-text search (MySQL and Postgres, with a full-text index; on Postgres ordered by relevance; SQLite through an FTS5 table named in the options). An integer query also finds the model with that key. shouldBeSearchable() is not applied, and withQuery() filters the search query itself. | | meilisearch | MEILISEARCH_HOST, MEILISEARCH_KEY. Filters and sort need the fields in filterableAttributes / sortableAttributes: search.meilisearch.indexSettings and search:sync-index-settings. | | typesense | TYPESENSE_HOST, TYPESENSE_PORT, TYPESENSE_PROTOCOL, TYPESENSE_API_KEY. The collection is made on the first write from search.typesense.modelSettings[Model].collectionSchema (or the model's typesenseCollectionSchema()); searchParameters.query_by names the searched fields; keys are strings (id: String(this.id)). | | algolia | ALGOLIA_APP_ID, ALGOLIA_SECRET. Sorting is a replica index (within()); SEARCH_IDENTIFY=true sends the searching user's IP and id. | | turbopuffer | TURBOPUFFER_API_KEY, TURBOPUFFER_REGION. search.turbopuffer.modelSettings[Model].searchableAttributes ({ title: 2, body: 1 }) ranks a query by BM25; one orderBy() sorts a search without a query. | | null | Indexes nothing, finds nothing. |

A model can use another engine than the default (searchableUsing() { return Search.engine('meilisearch') }), and an app can add its own: app(SearchManager).extend('mine', () => new MyEngine()), a class extending Engine.

Vector search (semantic(), hybrid()) is not supported yet.

Commands

| Command | | |---|---| | search:import <Model> [--fresh] [-c n] | Index every model (--fresh empties the index first) | | search:flush <Model> | Remove every model from the index | | search:queue-import <Model> [--min] [--max] [-c n] [--order] [--queue] | Index through the queue, one job per range of integer keys | | search:index <name\|Model> [-k key] | Create an index (and apply its settings) | | search:delete-index <name\|Model> | | | search:delete-all-indexes | Every index under the prefix (Meilisearch) | | search:sync-index-settings [--driver] | Apply indexSettings (Meilisearch, Algolia) |

<Model> is a class of src/models (or a module's models folder).

Testing

SEARCH_DRIVER=collection searches without any service, on SQLite too; null finds nothing. Queue.fake() records the index jobs with search.queue on.

Config

bunx ranting publish @rantingjs/search --tag config copies src/config/search.ts: driver, prefix, queue, afterCommit, chunk (searchable, unsearchable: 500), softDelete, identify, jobs, and the meilisearch, typesense, algolia and turbopuffer settings.