recall.js - Embedded RAG system

Recall.js is long term memory for AI apps!

It is a tool for building RAG (Retrieval-augmented generation) in a form of JavaScript library and command line utility focused on speed, ease of use and embeddability.

It is versatile and you don't have to use it exclusively for RAG, use it for generic Semantic Search, as expert memory for your AI app, as a recommendation system, there are so many possibilities...

Recall.js supports multilingual embeddings out of the box so you can add data in one language and then query it in another.

Under the hood, recall.js uses sentence vector embeddings and a vector database to index and query your data. It is a light wrapper around local language models such as MiniLM-L12-v2 and CozoDB vector database.

Install

npm install @sjovanovic/recall.js

Usage

Console:

recall --add 'The quick brown fox jumps over the lazy dog|Fox|{"foo":"bar"}'
recall --query "Un animal saute par-dessus un autre animal" --limit 1

Warning: when this library is used for the first time, it will download a local language model MiniLM-L12-v2 which may take long time depending on your Internet connectivity. Please be patient.

Below is the same example in JavaScript:

import * as RECALL from '@sjovanovic/recall.js'

const testRecall = async () => {
    await RECALL.addBatch([
        {
            input: "The quick brown fox jumps over the lazy dog",
            result: "Fox and dog",
            data: { foo: "bar" }
        }
    ])

    // Semantic search query in different language (French) "Animal jumps over another animal"
    let response = await RECALL.searchText("Un animal saute par-dessus un autre animal", 1)
    console.log(response)
}
testRecall()

/*

response:

{
  "headers": [
    "dist",
    "result",
    "id",
    "data"
  ],
  "rows": [
    [
      0.5840495824813843, // vector similarity
      "Fox and dog",
      "08840189191373282",
      {
        "foo": "bar"
      }
    ]
  ]
}

*/

Options

Easy way to view all the options is via command line:

recall --help

Usage:
recall --query "Foo Bar"

Options:
--query "SEARCH_STRING"                - search
--limit 2                              - limit number of results (used with --query)
--add 'input|result|{"foo":"bar"}'     - add data
--remove 'id'                          - remove data
--nuke                                 - destroy database
--mcp                                  - run as MCP server
--db "FILE_NAME"                       - database file (SQLite)
--import "file.csv | file.tsv"         - import from CSV or TSV w/ columns: 1. input 2. result 3. and remaining columns are additional data
--input-header "foo"                   - when used with --import designates specific header column as input
--result-header "bar"                  - when used with --import designates specific header column as result
--json "FILE_NAME"                     - import from file which has one json object per line: {input:"", result:"", data:{}}

Note: when adding data recall will generate unique id automatically. To set custom id add it as a string property named "id" in the data object (i.e. {"id":"customID"}).

JavaScript API Reference

RECALL.config

Configuration object.

export const config = {
    VECTOR_SIZE: 384, // number of dimensions
    MODEL_NAME: 'Xenova/paraphrase-multilingual-MiniLM-L12-v2', // model to use 
    SHOW_ERRORS: true, // Show errors
    DB_FILE: join(PATH, 'vector.db'), // Path to the datbase file (SQLite file used by CozoDB)
    PATH: PATH // directory of recall.js
}

RECALL.getDb()

Returns reference to the CozoDB instance.

RECALL.getEmbeddings(text) -> Promise(Array)

Given text calculates the embeddings vector

RECALL.add(input, result, data={}) -> Promise(Object)

Add data. input is the sentence to get embeddings from. result is the string to show in the results. data is arbitrary object intended to hold related pieces of information and references. If data object contains id property it will be used as unique id of the record.

RECALL.addBatch(batch) -> Promise(Object)

Add data in batches (faster than using add repeteadely). batch is an Array that looks like this:

let batch = [{input:"", result:"", data:{}}]

RECALL.remove(id) -> Promise(Object)

Remove data by id. id is a string.

RECALL.searchText(text, numResults = 5) -> Promise(Object)

Query the vector database. Accepts query text and number of results to return.

RECALL.nuke()

Deletes the database.

RECALL.importFromJSONStream(fileName) -> Promise(object)

Imports from readable stream or file which consists of JSON objects, one per line. e.g.

{input:"one", result:"one result", data:{"id":"123"}}
{input:"", result:"", data:{}}
...

This is the most efficient way to import data.

RECALL.importFromCSVorTSV(fileName, inputHeader=null, resultHeader=null) -> Promise()

Imports from CSV or TSV file. By default fist column is used as input, second as result and remaining columns are put in the data object. If inputHeader is specified, function will try to find the column by that name and use it as input. If resultHeader is specified, function will try to find the column by that name and use it as result.

RECALL.mcp() -> Promise()

(Experimental) Runs MCP server and makes the results available when mentioning Recall search in the prompt. Currently only supports STDIO.