@tricoteuses/meilisearch

Generic Meilisearch sync and hybrid/federated search engine shared across Tricoteuses products (tricoteuses-api-parlement, tricoteuses-api-legifrance, and future products).

The package carries no product-specific configuration — no stop words, no synonyms, no embedding provider, no data model. Everything domain-specific (which indexes exist, what fields are searchable, how documents are fetched) is provided by the consumer.

Install

npm install @tricoteuses/meilisearch

Client

import { createMeilisearchClient } from "@tricoteuses/meilisearch"

const client = createMeilisearchClient({
  host: process.env.MEILISEARCH_HOST!,
  apiKey: process.env.MEILISEARCH_API_KEY,
  // Optional: index settings merged as defaults into every sync service built from this client
  // (a service's own `settings` win on conflicting keys) — apply what's shared across every
  // index (tokenization, stop words...) once, here, instead of every index repeating it.
  defaultSettings: {
    nonSeparatorTokens: ["-"],
    stopWords: ["le", "la", "les" /* ... */],
  },
})

createMeilisearchClient returns { client, config }, not a bare Meilisearch instance — pass the whole thing (client above) as client to createSearchService/createMeilisearchSyncService, which unwrap it internally.

Sync engine

createMeilisearchSyncService builds a sync service for one Meilisearch index. It handles full rebuilds (via a temporary index + atomic swap) and incremental upserts/deletes, batching, and invalid-uid filtering — the only thing the consumer supplies is how to fetch a page of documents.

import { createMeilisearchSyncService } from "@tricoteuses/meilisearch"

const service = createMeilisearchSyncService("paragraphes", {
  client,
  batchSize: 10_000, // default: 1000
  primaryKey: "uid", // default: "uid"
  settings: {
    searchableAttributes: ["texte", "orateur"],
    filterableAttributes: ["dossierRefUid", "chambre"],
    sortableAttributes: ["dateSeance"],
    rankingRules: ["words", "typo", "proximity", "attributeRank", "sort", "exactness"],
    // Real Meilisearch settings.embedders — the client's defaultEmbedderConfig (if any) is
    // merged into every entry here, this service's own fields winning on conflicting keys.
    embedders: {
      paragraphes_embedder: {
        source: "openAi",
        documentTemplate: "{{ doc.texte }}",
      },
    },
  },
  fetchBatch: async ({ cursor, batchSize, ids }) => {
    // ids is set for incremental upserts, cursor for a full-rebuild scan
    return prisma.paragraphe.findMany({
      where: ids ? { uid: { in: ids } } : undefined,
      take: batchSize,
      skip: cursor ? 1 : 0,
      cursor: cursor ? { uid: cursor } : undefined,
    })
  },
})

await service.init() // ensures the index exists
service.trackUpsert(["uid-1", "uid-2"])
service.trackDelete(["uid-3"])
await service.flush("incremental") // or "full" to rebuild the whole index instead

mode is a parameter of flush(), not a construction option — nothing else about the service varies by mode, so it's safe (and often desirable) to build the service once as a module-level singleton and call flush(mode) with whatever mode each run needs. flush() always clears its pending upsert/delete tracking before returning, success or failure, so reusing the same instance across runs never leaks stale ids into the next one.

A product with several indexes sharing one embedding provider typically puts the fields that don't vary per index (model, apiKey, binaryQuantized...) in the client's defaultEmbedderConfig, merged automatically into every entry of every service's own settings.embedders:

const client = createMeilisearchClient({
  host: process.env.MEILISEARCH_HOST!,
  defaultEmbedderConfig: {
    source: "openAi",
    model: "text-embedding-3-small",
    apiKey: process.env.OPENAI_API_KEY!,
    binaryQuantized: true,
  },
})

createMeilisearchSyncService("paragraphes", {
  client,
  settings: {
    embedders: {
      // `source` still has to be repeated here: settings.embedders is Meilisearch's own type
      // (Record<string, Embedder>), and every Embedder variant requires `source` — the merge
      // with defaultEmbedderConfig happens after TypeScript has already checked this literal.
      paragraphes_embedder: { source: "openAi", documentTemplate: "{{ doc.texte }}" },
    },
  },
  fetchBatch,
})

Search engine

createSearchService wraps single-index search, weighted/federated multi-index search, and result hydration (loading full rows from your own database once Meilisearch has ranked the hit ids).

import { createSearchService } from "@tricoteuses/meilisearch"

const searchService = createSearchService({
  client,
  // Expanded into the query before searching — your own vocabulary, not the package's
  synonyms: { senateur: ["sénateur", "sénatrice"] },
})

// Single index, optionally hybrid (semanticRatio requires an embedder on that index's settings)
const { hits, totalHits, totalPages } = await searchService.search({
  index: "paragraphes",
  query: "réforme des retraites",
  filters: { chambre: "AN", dateSeance: { gte: "2023-01-01" } },
  semanticRatio: 0.5,
})

// Sorted search (fields must be in the index's sortableAttributes); Date filter values are sent as ISO strings
await searchService.search({
  index: "textes",
  query: "loi",
  filters: { datePublication: { gte: new Date("2024-01-01") } },
  sort: [{ field: "datePublication", direction: "desc" }],
})

// Federated global search across several indexes, weighted and thresholded per index
const globalHits = await searchService.multisearch(
  "dupont",
  [
    { index: "acteurs", attributesToSearchOn: ["nom", "prenom"], weight: 200, rankingScoreThreshold: 0.5 },
    { index: "dossiers", attributesToSearchOn: ["titre"], weight: 20, rankingScoreThreshold: 0.5 },
  ],
  { federated: { limit: 20 } },
)

// Search then load full rows from your own database, matched back to the search hits
const { hits: rows } = await searchService.searchAndHydrate({
  index: "acteurs",
  query: "dupont",
  hydrate: (ids) => prisma.acteur.findMany({ where: { uid: { in: ids } } }),
})

Optional: git-diff sync orchestrator

If your product's data is delivered as git-tracked datasets — one file per document, published by an upstream producer repo (as with tricoteuses-assemblee/tricoteuses-senat/tricoteuses-legifrance) — @tricoteuses/meilisearch/git-sync decides which document ids changed since the last sync and drives a sync service accordingly. It has nothing to do with Meilisearch itself; consumers who detect changes another way (a timestamp column, a queue, a webhook) can ignore this entirely.

import { describeDataset, describeDatasets, syncIndex, type MigrationStore } from "@tricoteuses/meilisearch/git-sync"

// Bring your own persistence for "what commit hash did we last sync from" —
// this package has no opinion on your schema.
const migrationStore: MigrationStore = {
  getLastSyncedHash: (indexName, dataset) =>
    prisma.searchSyncMigration
      .findFirst({ where: { indexName, dataset, status: "success" }, orderBy: { executedAt: "desc" } })
      .then((row) => row?.commitHash ?? null),
  recordSyncMigration: (data) => prisma.searchSyncMigration.create({ data }).then(() => undefined),
}

// You decide when/how "your product's options" (a data directory, a legislature filter...) turn
// into dataset descriptors — the package doesn't need to know their shape. describeDataset/
// describeDatasets standardize "<baseDir>/<relativePath>, or untracked if baseDir isn't set" —
// the mechanical part of a git-tracked dataset.
const datasets = [
  ...describeDatasets(myUpstreamDatasetList, assembleeDataDir, (d) => ({
    name: d.name,
    relativePath: `${d.name}_nettoye`,
  })),
  describeDataset("comptes-rendus", senatDataDir, "seances"),
]

await syncIndex("paragraphes", {
  migrationStore,
  forceRebuild: false,
  config: {
    incremental: true,
    service: createMeilisearchSyncService("paragraphes", { client, fetchBatch, settings }),
    datasets,
    // Optional: translate file-basename uids from the git diff into actual document uids
    resolveUids: async (debatUids) => {
      const rows = await prisma.paragraphe.findMany({ where: { debatRefUid: { in: debatUids } }, select: { uid: true } })
      return rows.map((r) => r.uid)
    },
  },
})

To sync several indexes in one run, syncIndexes loops over syncIndex, keeps going when one index fails so it doesn't block the rest, and reports which ones failed — deciding the process exit code stays up to the caller:

import { syncIndexes } from "@tricoteuses/meilisearch/git-sync"

const { failures } = await syncIndexes(["acteurs", "paragraphes"], {
  configs: { acteurs: acteursConfig, paragraphes: paragraphesConfig },
  migrationStore,
  forceRebuild: false,
})

if (failures.length > 0) process.exit(1)

defineIndexes builds that { names, configs } pair (in one ordered, single-source-of-truth list) instead of hand-maintaining a separate name array and a keyed Record:

import { defineIndexes, syncIndexes } from "@tricoteuses/meilisearch/git-sync"

const { names, configs } = defineIndexes([
  { name: "acteurs", service: acteursService, incremental: false, datasets: acteursDatasets },
  { name: "paragraphes", service: paragraphesService, incremental: true, datasets: paragraphesDatasets },
])

const { failures } = await syncIndexes(names, { configs, migrationStore })

What this package does not provide

• No stop words, synonyms, or embedding provider — all consumer-supplied via settings/synonyms/embedders.
• No Prisma schema or persistence layer — MigrationStore is an interface you implement against your own data model.
• No HTTP routes/controllers, no CLI, no per-domain index configuration. Those stay in each consuming product.