Overview

Kestrel is for teams that want globally unique, time-sortable IDs in a Redis-backed system and want those IDs to be useful later in analytics.

It’s a good fit for:

• API + data platform teams: generate IDs at write-time, then decode them in your warehouse/ETL jobs for time-based partitioning and debugging.
• Distributed systems: multiple app instances can generate IDs without database round-trips, while preserving ordering by creation time.

What you get:

• K-sortable 64-bit IDs: IDs sort by creation time (milliseconds since a custom epoch).
• Embedded metadata: each ID encodes:
  • timestamp (ms)
  • logical shard id
  • per-millisecond sequence (0–4095)
• Decoding utilities: Kestrel.decodeId() / Kestrel.decodeIds() to extract timestamp/shard/sequence for:
  • data warehousing (e.g., derive created_at, partition keys, time-window rollups)
  • ETL/observability (e.g., detect hot shards, analyze write bursts, investigate gaps)

See examples/ for decode-focused scripts you can drop into warehouse/ETL workflows.

Getting started

Prerequisites

• Node.js v20+ (ES modules support required)
• Redis v7+ (standalone), or Redis Sentinel / Redis Cluster (optional modes)
• Docker + Docker Compose (for the included local Sentinel/Cluster test stacks)

# Initialize the repo (if applicable)
./.scripts/init.sh

Installation

npm install @dreamystify/kestrel

To build the package locally with esbuild, run:

ahoy build

ES Modules Support

This package is built as an ES module and requires Node.js v18+ with ES modules support. When using this package:

• Use import statements instead of require()
• Ensure your package.json has "type": "module" or use .mjs file extensions
• The package exports are optimized for ES module resolution

Usage

The Kestrel library supports three connection modes: standalone, Sentinel, and Cluster. Below are examples for each mode.

Standalone Mode

import { Kestrel } from '@dreamystify/kestrel';

(async () => {
  // Initialize Kestrel with standalone Redis
  const kestrel = await Kestrel.initialize({
    host: 'localhost',
    port: 6379,
    username: 'yourUsername', // if using authentication
    password: 'yourPassword', // if using authentication
    database: 0,
  });

  // Generate a single unique ID
  const id = await kestrel.getId();
  console.log('Generated ID:', id);

  // Generate a batch of 3 IDs
  const ids = await kestrel.getIds(3);
  console.log('Generated IDs:', ids);
})();

Sentinel Mode

import { Kestrel } from '@dreamystify/kestrel';

(async () => {
  // Initialize Kestrel using Redis Sentinel
  const kestrel = await Kestrel.initialize({
    sentinels: [
      { host: 'sentinel1', port: 26379 },
      { host: 'sentinel2', port: 26379 },
    ],
    name: 'mymaster', // name of your master instance
    username: 'yourUsername', // for the master
    password: 'yourPassword', // for the master
    sentinelUsername: 'sentinelUser', // if your Sentinel requires authentication
    sentinelPassword: 'sentinelPassword', // if your Sentinel requires authentication
    database: 0,
  });

  const id = await kestrel.getId();
  console.log('Generated ID:', id);
})();

Cluster Mode

import { Kestrel } from '@dreamystify/kestrel';

(async () => {
  // Initialize Kestrel using Redis Cluster
  const kestrel = await Kestrel.initialize({
    clusterNodes: [
      { host: 'redis-cluster-node1', port: 6379 },
      { host: 'redis-cluster-node2', port: 6379 },
      { host: 'redis-cluster-node3', port: 6379 },
    ],
    username: 'yourUsername', // for cluster authentication
    password: 'yourPassword', // for cluster authentication
    database: 0,
  });

  const id = await kestrel.getId();
  console.log('Generated ID:', id);
})();

Sharding Configuration

import { Kestrel } from '@dreamystify/kestrel';

(async () => {
  // Initialize Kestrel with a custom shard configuration (for standalone mode)
  const kestrel = await Kestrel.initialize({
    host: 'localhost',
    port: 6379,
    username: 'yourUsername',
    password: 'yourPassword',
    database: 0,
    // Optionally set a fixed shard ID via environment variable:
    // KESTREL_SHARD_ID_KEY: '{kestrel}-shard-id',
    // KESTREL_SHARD_ID: '1',
  });

  const id = await kestrel.getId();
  console.log('Generated ID:', id);
})();

In Cluster mode, the library automatically assigns shard IDs to master nodes based on the cluster topology. You can later query each node (using redis-cli) to see the assigned shard IDs:

docker run -it --rm --network redis_cluster redis:7.0.2 redis-cli -a yourPassword --cluster call redis-cluster-node1:6379 GET '{kestrel}-shard-id'

Decoding IDs

Kestrel IDs can be decoded back into their component parts, allowing you to extract information like when the ID was created, which shard generated it, and the sequence number.

import { Kestrel } from '@dreamystify/kestrel';

(async () => {
  const kestrel = await Kestrel.initialize({
    host: 'localhost',
    port: 6379,
    username: 'yourUsername',
    password: 'yourPassword',
  });

  // Generate an ID
  const id = await kestrel.getId();

  // Decode a single ID
  const decoded = Kestrel.decodeId(id);
  console.log('Timestamp (ms since Unix epoch):', decoded.timestampMs);
  console.log('Timestamp (ms since custom epoch):', decoded.timestamp);
  console.log('Logical Shard ID:', decoded.logicalShardId); // 0-1023
  console.log('Sequence:', decoded.sequence); // 0-4095
  console.log('Created At:', decoded.createdAt); // JavaScript Date object

  // Decode multiple IDs
  const ids = await kestrel.getIds(5);
  const decodedIds = Kestrel.decodeIds(ids);
  decodedIds.forEach((d, index) => {
    console.log(`ID ${index + 1}:`);
    console.log(`  Created: ${d.createdAt.toISOString()}`);
    console.log(`  Shard: ${d.logicalShardId}, Sequence: ${d.sequence}`);
  });
})();

The decodeId method accepts IDs as bigint, string, or number, making it easy to decode IDs from various sources (databases, APIs, etc.).

Error Handling

Kestrel emits events for errors. It is designed to let your application handle logging and error processing in a way that suits your needs. For example:

kestrel.on('error', error => {
  console.error('Kestrel error:', error.error);
});

kestrel.on('connect', () => {
  console.log('Kestrel connected to Redis');
});

Events

| Event Constant | Event String | Description | | ------------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------- | | CLIENT_CREATED | clientCreated | Emitted when the Redis client instance is successfully created. | | CONNECTED | connected | Emitted when the client has successfully connected to Redis. | | SCRIPT_LOADED | scriptLoaded | Emitted when a Lua script is successfully loaded on a Redis node. | | NODE_ADDED | nodeAdded | Emitted when a new node is detected in a cluster. | | NODE_REMOVED | nodeRemoved | Emitted when a node is removed from a cluster. | | CLUSTER_NODE_ADDED | +node | Emitted when a new cluster node is added (internal cluster topology event). | | CLUSTER_NODE_REMOVED | -node | Emitted when a cluster node is removed (internal cluster topology event). | | ERROR | error | Emitted when an error occurs within the Kestrel library. | | Redis Connection Events | | | | CONNECT | connect | Emitted when a connection is established. | | CONNECTING | connecting | Emitted when the client is attempting to establish a connection. | | RECONNECTING | reconnecting | Emitted when the client is attempting to reconnect after a disconnect or error. | | DISCONNECTED | disconnected | Emitted when the client has been disconnected. | | WAIT | wait | Emitted when the client is waiting (typically during retry/backoff). | | READY | ready | Emitted when the client is ready to accept commands. | | CLOSE | close | Emitted when the connection is closed. | | END | end | Emitted when the connection has ended. | | RECONNECTED | reconnected | Emitted when the client has successfully reconnected. | | RECONNECTION_ATTEMPTS_REACHED | reconnectionAttemptsReached | Emitted when the maximum number of reconnection attempts is reached and no further retries occur. |

Testing

# Start the testing environment
ahoy start

# Run the tests with coverage
ahoy test

# Run tests with coverage reporting
ahoy coverage

# Stop the testing environment
ahoy stop

Testing Sentinel and Cluster

# Start Redis containers (standalone + sentinel + cluster) and wait until they're ready
ahoy start

# Run the Sentinel harness (uses tests/sentinel.js)
ahoy sentinel

# Run the Cluster harness (uses tests/cluster.js)
ahoy cluster

# Inspect harness container logs (if needed)
ahoy logs

# Stop containers
ahoy stop

Kudos

The project was inspired by Icicle, just setup for node.js.