@keyv/bigmap

Bigmap for Keyv

Features

• Based on the Map interface and uses the same API.
• Lightweight and easy to use.
• Scales to past the 17 million key limit of a regular Map.
• Ability to use your own hash function.
• Built in Typescript and Generics for type safety.
• Used in @cacheable/memory for scalable in-memory caching.
• Maintained regularly with a focus on performance and reliability.

Table of Contents

• Features
• Installation
• Overview
• Basic Usage
• Custom Store Size
• Custom Hash Function
• Iteration
  • For...of Loop
  • forEach
  • Keys, Values, and Entries
• Events
• Advanced Features
  • Type Safety with Generics
  • Large-Scale Data
• Using with Keyv
  • createKeyv
  • With Custom Options
  • Type Safety
• Integration with Keyv Ecosystem
• API
  • Constructor
  • Properties
  • Methods
    • set
    • get
    • has
    • delete
    • clear
    • forEach
    • keys
    • values
    • entries
    • Symbol.iterator
    • getStore
    • getStoreMap
    • initStore
• Types
• StoreHashFunction
• BigMapEvents
• defaultHashFunction(key, storeSize)
• Benchmark
• Migrating to v6
• Contributing
• License

Installation

npm install --save keyv @keyv/bigmap

Overview

BigMap is a scalable Map implementation that overcomes JavaScript's built-in Map limit of approximately 17 million entries. It uses a distributed hash approach with multiple internal Map instances.

Basic Usage

import { BigMap } from '@keyv/bigmap';

// Create a new BigMap
const bigMap = new BigMap<string, number>();

// Set values
bigMap.set('key1', 100);
bigMap.set('key2', 200);

// Get values
const value = bigMap.get('key1'); // 100

// Check if key exists
bigMap.has('key1'); // true

// Delete a key
bigMap.delete('key1'); // true

// Get size
console.log(bigMap.size); // 1

// Clear all entries
bigMap.clear();

Custom Store Size

By default, BigMap uses 2 internal Map instances. You can configure this:

const bigMap = new BigMap<string, number>({ storeSize: 10 });

Changing storeSize after initialization keeps every entry: BigMap moves each one into the internal Map its key maps to at the new size. That takes one pass over all entries, so set storeSize in the constructor for a large map.

Custom Hash Function

Provide your own hash function for key distribution:

const customHashFunction = (key: string, storeSize: number) => {
  return key.length % storeSize;
};

const bigMap = new BigMap<string, string>({
  storeHashFunction: customHashFunction
});

You can also set storeHashFunction later. BigMap then moves every existing entry into the Map the new function picks, so get(), has() and delete() keep finding them. If the function throws while entries are moved, the setter rethrows and the map is left as it was.

Using Hashery for Hash Functions

Hashery is a powerful hashing library that provides multiple hash algorithms. You can use it for better key distribution and it is available as an export:

import { BigMap, Hashery } from '@keyv/bigmap';

const hashery = new Hashery();

// Using Hashery's toNumberSync for deterministic key distribution
const bigMap = new BigMap<string, string>({
  storeHashFunction: (key: string, storeSize: number) => {
    return hashery.toNumberSync(key, { min: 0, max: storeSize - 1 });
  }
});

// You can also use different algorithms
const hasheryFnv1 = new Hashery({ defaultAlgorithmSync: 'fnv1' });

const bigMapWithFnv1 = new BigMap<string, string>({
  storeHashFunction: (key: string, storeSize: number) => {
    return hasheryFnv1.toNumberSync(key, { min: 0, max: storeSize - 1 });
  }
});

Hashery supports multiple synchronous hash algorithms:

• djb2 - Fast hash function (default)
• fnv1 - Excellent distribution for hash tables
• murmer - MurmurHash algorithm
• crc32 - Cyclic Redundancy Check

Iteration

BigMap supports all standard Map iteration methods:

For...of Loop

const bigMap = new BigMap<string, number>();
bigMap.set('a', 1);
bigMap.set('b', 2);

for (const [key, value] of bigMap) {
  console.log(key, value);
}

forEach

bigMap.forEach((value, key) => {
  console.log(key, value);
});

// With custom context
const context = { sum: 0 };
bigMap.forEach(function(value) {
  this.sum += value;
}, context);

Keys, Values, and Entries

// Iterate over keys
for (const key of bigMap.keys()) {
  console.log(key);
}

// Iterate over values
for (const value of bigMap.values()) {
  console.log(value);
}

// Iterate over entries
for (const [key, value] of bigMap.entries()) {
  console.log(key, value);
}

Events

BigMap extends Hookified, so it emits events on mutating operations. Subscribe with on (or once) using the BigMapEvents enum (the values are also plain strings such as "set"). Listeners are invoked synchronously, and BigMapOptions also accepts any Hookified options.

| Event | Enum | Payload | Description | |-------------------|----------------------|----------------|------------------------------------------------------------------------------| | set | BigMapEvents.SET | (key, value) | Emitted after a value is set. | | delete | BigMapEvents.DELETE| (key) | Emitted after an existing entry is removed. Not emitted when the key was absent. | | clear | BigMapEvents.CLEAR | (none) | Emitted after all entries are cleared by clear(). |

import { BigMap, BigMapEvents } from '@keyv/bigmap';

const bigMap = new BigMap<string, number>();

bigMap.on(BigMapEvents.SET, (key, value) => {
  console.log('set', key, value);
});

bigMap.on(BigMapEvents.DELETE, (key) => {
  console.log('deleted', key);
});

bigMap.on(BigMapEvents.CLEAR, () => {
  console.log('cleared');
});

bigMap.set('key1', 100); // logs: set key1 100
bigMap.delete('key1');   // logs: deleted key1
bigMap.clear();          // logs: cleared

Note: When Hookified's eventLogger option is enabled, event arguments — including the value passed to set — are serialized with JSON.stringify for logging. Values that cannot be JSON-serialized, such as BigInt or circular objects, will cause the operation to throw while logging. Leave eventLogger unset (the default) if you need to store such values.

Advanced Features

Type Safety with Generics

interface User {
  id: number;
  name: string;
}

const userMap = new BigMap<string, User>();
userMap.set('user1', { id: 1, name: 'Alice' });

Large-Scale Data

BigMap is designed to handle millions of entries:

const bigMap = new BigMap<string, number>({ storeSize: 16 });

// Add 20+ million entries without hitting Map limits
for (let i = 0; i < 20000000; i++) {
  bigMap.set(`key${i}`, i);
}

console.log(bigMap.size); // 20000000

Using with Keyv

BigMap can be used as a storage adapter for Keyv, providing a scalable in-memory store with TTL support.

createKeyv

The createKeyv function creates a Keyv instance with BigMap as the storage adapter.

Parameters:

• options (optional): BigMap configuration options
  • storeSize (number): Number of internal Map instances. Default: 2
  • storeHashFunction (StoreHashFunction): Custom hash function for key distribution

Returns: Keyv instance with BigMap adapter

Example:

import { createKeyv } from '@keyv/bigmap';

// Basic usage
const keyv = createKeyv();

// Set with TTL (in milliseconds)
await keyv.set('user:123', { name: 'Alice', age: 30 }, 60000); // Expires in 60 seconds

// Get value
const user = await keyv.get('user:123');
console.log(user); // { name: 'Alice', age: 30 }

// Check if key exists
const exists = await keyv.has('user:123');

// Delete key
await keyv.delete('user:123');

// Clear all keys
await keyv.clear();

With Custom Options

import { createKeyv } from '@keyv/bigmap';

// Create with custom store size for better performance with millions of keys
const keyv = createKeyv({ storeSize: 16 });

// With custom hash function
const keyv = createKeyv({
  storeSize: 8,
  storeHashFunction: (key, storeSize) => {
    // Custom distribution logic
    return key.length % storeSize;
  }
});

Type Safety

import { createKeyv } from '@keyv/bigmap';

interface Product {
  id: string;
  name: string;
  price: number;
}

const keyv = createKeyv<string, Product>();

await keyv.set('product:1', {
  id: '1',
  name: 'Laptop',
  price: 999
});

const product = await keyv.get<Product>('product:1');

Integration with Keyv Ecosystem

BigMap works seamlessly with the Keyv ecosystem:

import { createKeyv } from '@keyv/bigmap';

const cache = createKeyv({ storeSize: 16 });

// Use with namespaces
const users = cache.namespace('users');
const products = cache.namespace('products');

await users.set('123', { name: 'Alice' });
await products.set('456', { name: 'Laptop' });

// Iterate over keys
for await (const [key, value] of cache.iterator()) {
  console.log(key, value);
}

API

Constructor

new BigMap<K, V>(options?)

Creates a new BigMap instance.

Parameters:

• options (optional): Configuration options
  • storeSize (number): Number of internal Map instances to use. Default: 2. Must be at least 1.
  • storeHashFunction (StoreHashFunction): Custom hash function for key distribution. Default: defaultHashFunction

Example:

const bigMap = new BigMap<string, number>();
const customBigMap = new BigMap<string, number>({
  storeSize: 10,
  storeHashFunction: (key, storeSize) => key.length % storeSize
});

Properties

| Property | Type | Access | Description | |----------|------|--------|-------------| | size | number | Read-only | Gets the total number of entries in the BigMap. | | storeSize | number | Read/Write | Gets or sets the number of internal Map instances. Setting it moves every entry into the new layout, so no entries are lost. Default: 2 | | storeHashFunction | StoreHashFunction \| undefined | Read/Write | Gets or sets the hash function used for key distribution. Setting it moves every entry into the Map the new function picks. | | store | Array<Map<K, V>> | Read-only | Gets the internal array of Map instances. |

Examples:

const bigMap = new BigMap<string, number>();

// size property
bigMap.set('key1', 100);
console.log(bigMap.size); // 1

// storeSize property
console.log(bigMap.storeSize); // 2 (default)
bigMap.storeSize = 8; // Changes size and keeps every entry

// storeHashFunction property
bigMap.storeHashFunction = (key, storeSize) => key.length % storeSize;

// store property
console.log(bigMap.store.length); // 8

Methods

set

Sets the value for a key in the map.

Parameters:

• key (K): The key to set
• value (V): The value to associate with the key

Returns: BigMap<K, V> - The BigMap instance, enabling method chaining (matches the native Map.set API).

Example:

bigMap.set('user123', { name: 'Alice' });

// set() returns the BigMap instance, so calls can be chained
bigMap.set('a', { name: 'Alice' }).set('b', { name: 'Bob' });

get

Gets the value associated with a key.

Parameters:

• key (K): The key to retrieve

Returns: V | undefined - The value, or undefined if not found

Example:

const value = bigMap.get('user123');

has

Checks if a key exists in the map.

Parameters:

• key (K): The key to check

Returns: boolean - True if the key exists, false otherwise

Example:

if (bigMap.has('user123')) {
  console.log('User exists');
}

delete

Deletes a key-value pair from the map.

Parameters:

• key (K): The key to delete

Returns: boolean - True if the entry was deleted, false if the key was not found

Example:

const deleted = bigMap.delete('user123');

clear

Removes all entries from the map.

Returns: void

Example:

bigMap.clear();
console.log(bigMap.size); // 0

forEach

Executes a provided function once for each key-value pair.

Parameters:

• callbackfn (function): Function to execute for each entry
  • value (V): The value of the current entry
  • key (K): The key of the current entry
  • map (Map<K, V>): The BigMap instance
• thisArg (optional): Value to use as this when executing the callback

Returns: void

Example:

bigMap.forEach((value, key) => {
  console.log(`${key}: ${value}`);
});

// With custom context
const context = { total: 0 };
bigMap.forEach(function(value) {
  this.total += value;
}, context);

keys

Returns an iterator of all keys in the map.

Returns: IterableIterator<K>

Example:

for (const key of bigMap.keys()) {
  console.log(key);
}

values

Returns an iterator of all values in the map.

Returns: IterableIterator<V>

Example:

for (const value of bigMap.values()) {
  console.log(value);
}

entries

Returns an iterator of all key-value pairs in the map.

Returns: IterableIterator<[K, V]>

Example:

for (const [key, value] of bigMap.entries()) {
  console.log(key, value);
}

Symbol.iterator

Returns an iterator for the map (same as entries()). Enables for...of loops.

Returns: IterableIterator<[K, V]>

Example:

for (const [key, value] of bigMap) {
  console.log(key, value);
}

getStore

Gets the internal Map instance for a specific key.

Parameters:

• key (K): The key to find the store for

Returns: Map<K, V> - The internal Map instance

Example:

const store = bigMap.getStore('user123');

getStoreMap

Gets the internal Map instance at a specific index.

Parameters:

• index (number): The index of the Map to retrieve (0 to storeSize - 1)

Returns: Map<K, V> - The Map at the specified index

Throws: Error if index is out of bounds

Example:

const firstMap = bigMap.getStoreMap(0);

initStore

Initializes the internal store with empty Map instances. Called automatically during construction.

Returns: void

Types

StoreHashFunction

Type definition for custom hash functions.

type StoreHashFunction = (key: string, storeSize: number) => number;

Parameters:

• key (string): The key to hash (converted to string)
• storeSize (number): The total number of stores

Returns: number - The index of the store to use (0 to storeSize - 1)

BigMapEvents

Enum of the event names emitted by BigMap. See Events for payloads and usage.

enum BigMapEvents {
  SET = 'set',
  DELETE = 'delete',
  CLEAR = 'clear',
}

defaultHashFunction

The built-in hash function used for distributing keys across stores. It uses a fast multiplicative hash with XOR mixing to produce a bucket index. You can override it via the storeHashFunction option.

Example:

import { defaultHashFunction } from '@keyv/bigmap';

const index = defaultHashFunction('myKey', 2);

Benchmark

This benchmark compares BigMap against the native JavaScript Map for combined set and get operations using 10,000 iterations of randomly generated data. BigMap trades raw single-threaded speed for the ability to scale past the ~16.7 million entry limit of a native Map by distributing keys across multiple internal Map instances via a hash function. For most workloads under that limit, native Map will be faster; BigMap is designed for scenarios where you need to go beyond it.

| name | summary | ops/sec | time/op | margin | samples | |--------------------|:---------:|----------:|----------:|:--------:|----------:| | Map set / get | 🥇 | 19M | 64ns | ±0.02% | 16M | | BigMap set / get | -23% | 14M | 91ns | ±0.03% | 11M |

If you want to see comparable performance just set the storeSize: 1 and it is -1% off from native Map.

Migrating to v6

Keyv v5 used @keyv/bigmap 1.x. Changes in v6:

• Events. BigMap now emits set, delete and clear through its Hookified base, and the new BigMapEvents enum names them. In 1.x, BigMap extended Hookified but emitted no events. See Events.
• MapInterfacee renamed. The exported type is now MapInterface.
• set() returns the BigMap. Calls chain like Map.set. In 1.x, set() returned the internal Map that held the key.
• Default storeSize is 2. It was 4 in 1.x. Pass storeSize: 4 to keep the old layout.
• Hashing. defaultHashFunction is now a built-in hash instead of Hashery, so keys land in different internal Maps. A custom storeHashFunction now receives the real storeSize (1.x passed storeSize - 1), and an out-of-range result is wrapped into range instead of throwing. This matters only if you read store, getStore() or getStoreMap() directly.
• Changing storeSize or storeHashFunction keeps your entries. Both setters move every entry into the internal Map its key now maps to. In 1.x, setting storeSize removed every entry, and setting storeHashFunction left entries where get(), has() and delete() couldn't find them, while size still counted them and set() added a second copy. If you set storeSize to empty the map, call clear() as well.
• Dependencies. hashery moved from ^1.4 to ^3 and is still re-exported as Hashery. hookified moved from ^1.15 to ^3; if you pass Hookified options to BigMap, note that logger is now eventLogger.
• createKeyv still returns a Keyv instance backed by a new BigMap. Keyv v6 wraps the BigMap in its KeyvMemoryAdapter, which handles TTL and namespaces, so the BigMap itself is at keyv.store.store.

Contributing

Please see our contributing guide.

License

MIT © Jared Wray