npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

react-embeddings-search

v0.0.3

Published

A lightweight and efficient vector database for storing and searching text embeddings in the browser's local storage. The package uses OpenAI's API or Transformer.js models to generate embeddings for text documents and provides functionality for similarit

Downloads

4

Readme

Vector Storage

Forked from vector-storage with patches to allow for Transformer.js embeddings.

Vector Storage is a lightweight and efficient vector database that stores document vectors in the browser's IndexedDB. This package allows you to perform semantic similarity searches on text documents using vector embeddings. Semantic search refers to the ability to understand the meaning and context of text documents and queries, enabling more accurate and relevant search results. Vector Storage leverages OpenAI or Transformer.js embeddings (see compatible models here) to convert text documents into vectors and provides an interface for searching similar documents based on cosine similarity.

Features

  • Store and manage document vectors in IndexedDB
  • Perform similarity searches on text documents
  • Filter search results based on metadata or text content
  • Automatically manage storage size and remove least recently used documents when space limit is reached

Cosine Similarity Algorithm

Cosine similarity is a measure of similarity between two non-zero vectors in an inner product space. It is defined as the cosine of the angle between the two vectors. The cosine similarity value ranges from -1 to 1, where 1 indicates complete similarity, 0 indicates no similarity, and -1 indicates complete dissimilarity.

In this package, cosine similarity is used to measure the similarity between document vectors and the query vector. The cosine similarity score is calculated using the dot product of the vectors, divided by the product of their magnitudes.

LRU Mechanism

The Least Recently Used (LRU) mechanism is used to manage the storage size and automatically remove documents when the storage size exceeds the specified limit. Documents are sorted by their hit counter (ascending) and then by their timestamp (ascending). Documents with the lowest hit count and oldest timestamps are removed first until the storage size is below the limit.

Installation

Install the package using npm:

npm i vector-storage

Usage

Here is a basic example of how to use the VectorStorage class:

import { VectorStorage } from "vector-storage";

// Create an instance of VectorStorage
const vectorStore = new VectorStorage({ openAIApiKey: "your-openai-api-key" });

// Add a text document to the store
await vectorStore.addText("The quick brown fox jumps over the lazy dog.", {
  category: "example",
});

// Perform a similarity search
const results = await vectorStore.similaritySearch({
  query: "A fast fox leaps over a sleepy hound.",
});

// Display the search results
console.log(results);

API

VectorStorage

The main class for managing document vectors in IndexedDB.

constructor(options: IVSOptions)

Creates a new instance of VectorStorage.

options: An object containing the following properties:

interface IVSOptions {
  openAIApiKey: string; // The OpenAI API key used for generating embeddings.
  maxSizeInMB?: number; // The maximum size of the storage in megabytes. Defaults to 2GB
  debounceTime?: number; // The debounce time in milliseconds for saving to IndexedDB. Defaults to 0.
  openaiModel?: string; // The OpenAI model used for generating embeddings.
}

addText(text: string, metadata: object): Promise

Adds a text document to the store and returns the created document.

  • text: The text content of the document.
  • metadata: An object containing metadata associated with the document.

addTexts(texts: string[], metadatas: object[]): Promise<IVSDocument[]>

Adds multiple text documents to the store and returns an array of created documents.

  • texts: An array of text contents for the documents.
  • metadatas: An array of metadata objects associated with the documents.

similaritySearch(params: ISimilaritySearchParams): Promise<IVSDocument[]>

Performs a similarity search on the stored documents and returns an array of matching documents.

params: An object containing the following properties:

  • query: The query text or vector for the search.
  • k (optional): The number of top results to return (default: 4).
  • filterOptions (optional): An object specifying filter criteria for the search.

IVSDocument Interface

The IVSDocument interface represents a document object stored in the vector database. It contains the following properties:

interface IVSDocument {
  hits?: number; // The number of hits (accesses) for the document. Omit if the value is 0.
  metadata: object; // The metadata associated with the document for filtering.
  text: string; // The text content of the document.
  timestamp: number; // The timestamp indicating when the document was added to the store.
  vectorMag: number; // The magnitude of the document vector.
  vector: number[]; // The vector representation of the document.
}

Contributing

Contributions to this project are welcome! If you would like to contribute, please follow these steps:

  1. Fork the repository on GitHub.
  2. Clone your fork to your local machine.
  3. Create a new branch for your changes.
  4. Make your changes and commit them to your branch.
  5. Push your changes to your fork on GitHub.
  6. Open a pull request from your branch to the main repository.

Please ensure that your code follows the project's coding style and that all tests pass before submitting a pull request. If you find any bugs or have suggestions for improvements, feel free to open an issue on GitHub.

License

This project is licensed under the MIT License. See the LICENSE file for the full license text.

Copyright (c) Nitai Aharoni. All rights reserved.