Prisma Sharding

Lightweight database sharding library for Prisma with connection pooling, health monitoring, and CLI tools.

Installation

yarn add prisma-sharding
# or
npm install prisma-sharding

Don't forget to follow me on GitHub!

Step 1: Create Sharding Connection

// src/config/prisma.ts

import { PrismaSharding } from 'prisma-sharding';
import { PrismaClient } from '@/generated/prisma';
import { PrismaPg } from '@prisma/adapter-pg';

const sharding = new PrismaSharding<PrismaClient>({
  shards: [
    { id: 'shard_1', url: process.env.SHARD_1_URL! },
    { id: 'shard_2', url: process.env.SHARD_2_URL! },
  ],
  strategy: 'modulo', // 'modulo' | 'consistent-hash'
  createClient: (url) => {
    const adapter = new PrismaPg({ connectionString: url, max: 10 });
    return new PrismaClient({ adapter });
  },
});

await sharding.connect();

API

| Method | Description | | ---------------------------- | --------------------------------------------- | | getShard(key) | Get Prisma client for a given key | | getShardById(shardId) | Get Prisma client by shard ID | | getRandomShard() | Get random shard (for new records) | | findFirst(fn) | Search across all shards, return first result | | runOnAll(fn) | Execute on all shards | | getHealth() | Get health status of all shards | | connect() / disconnect() | Lifecycle methods |

Step 2: Create a User (Assign to a Shard)

New records should be created on a random shard for even distribution.

import { sharding } from '@/config/prisma';

const client = sharding.getRandomShard();

const user = await client.user.create({
  data: {
    email: '[email protected]',
    username: 'new_user',
  },
});

Step 3: Access User by ID (Shard Routing)

When you have a user ID, Prisma Sharding routes you to the correct shard automatically.

const userId = 'abc123';

const client = sharding.getShard(userId);

const user = await client.user.findUnique({
  where: { id: userId },
});

Important rule:

Once you get the shard client using a user ID, all future operations for that user must use this same client.

That includes:

• Reading user data
• Updating user data
• Creating related records (profiles, posts, settings, etc)

Every user belongs to exactly one shard. Their entire data lives on that shard only.

Do not switch shards or use a random shard for user related actions.

Always do this:

const client = sharding.getShard(userId);

This guarantees all user data stays on the correct shard and avoids cross shard bugs.

Step 4: Find User Without ID (Cross Shard Search)

If you do not have the user ID, search all shards in parallel. Use this only when necessary.

// Find user by email across ALL shards (parallel execution)
const { result: user, client } = await sharding.findFirst(async (c) =>
  c.user.findFirst({ where: { email } })
);

if (user && client) {
  // Continue operations on the found shard
  await client.user.update({
    where: { id: user.id },
    data: { lastLogin: new Date() },
  });
}

Step 5: Run on All Shards (Admin or Analytics)

// Get counts from all shards
const counts = await sharding.runOnAll(async (client) => client.user.count());
const totalUsers = counts.reduce((sum, count) => sum + count, 0);

// With detailed results (includes errors)
const results = await sharding.runOnAllWithDetails(async (client, shardId) => {
  return { shardId, count: await client.user.count() };
});

Health Monitoring

// Get health of all shards
const health = sharding.getHealth();
// Returns: [{ shardId, isHealthy, latencyMs, lastChecked, ... }]

// Get specific shard health
const shard1Health = sharding.getHealthByShard('shard_1');

Lifecycle

// Graceful shutdown
await sharding.disconnect();

// Check connection status
if (sharding.isConnected()) {
  // ...
}

CLI Tools

The package includes CLI tools for common sharding operations. No need to write custom scripts!

Setup

Add to your package.json:

{
  "scripts": {
    "db:studio:all": "prisma-sharding-studio",
    "migrate:shards": "prisma-sharding-migrate",
    "test:shards": "prisma-sharding-test"
  }
}

Environment Variables

SHARD_COUNT=3
SHARD_1_URL=postgresql://user:pass@host:5432/db1
SHARD_2_URL=postgresql://user:pass@host:5432/db2
SHARD_3_URL=postgresql://user:pass@host:5432/db3
SHARD_ROUTING_STRATEGY=modulo  # or consistent-hash
SHARD_STUDIO_BASE_PORT=51212   # optional, for studio

Commands

prisma-sharding-migrate

Push schema to all shards using prisma db push.

yarn migrate:shards

prisma-sharding-studio

Start Prisma Studio for all shards on sequential ports.

yarn db:studio:all
# Opens shard_1 on :51212, shard_2 on :51213, etc.

prisma-sharding-test

Test connections to all shards.

yarn test:shards

================================
📋 User Distribution Test
================================
Creating 24 test users across 3 shards...

User 1/24: "testuser_0" → shard_3
User 2/24: "testuser_1" → shard_1
User 3/24: "testuser_2" → shard_2
User 4/24: "testuser_3" → shard_3
User 5/24: "testuser_4" → shard_1
User 6/24: "testuser_5" → shard_2
User 7/24: "testuser_6" → shard_3
User 8/24: "testuser_7" → shard_1
User 9/24: "testuser_8" → shard_2
User 10/24: "testuser_9" → shard_3
User 11/24: "testuser_10" → shard_2
User 12/24: "testuser_11" → shard_1
User 13/24: "testuser_12" → shard_3
User 14/24: "testuser_13" → shard_2
User 15/24: "testuser_14" → shard_1
User 16/24: "testuser_15" → shard_3
User 17/24: "testuser_16" → shard_2
User 18/24: "testuser_17" → shard_1
User 19/24: "testuser_18" → shard_3
User 20/24: "testuser_19" → shard_2
User 21/24: "testuser_20" → shard_1
User 22/24: "testuser_21" → shard_3
User 23/24: "testuser_22" → shard_2
User 24/24: "testuser_23" → shard_1
✅ Created 24/24 test users

================================
📋 Read Verification
================================
✓ User "test_user_1770289330292_0" found on shard_3
✓ User "test_user_1770289330292_1" found on shard_1
✓ User "test_user_1770289330292_2" found on shard_2
✓ User "test_user_1770289330292_3" found on shard_3
✓ User "test_user_1770289330292_4" found on shard_1
Verified 5/5 users on correct shards
✅ Verify users exist on correct shards (136ms)

Configuration

| Option | Type | Default | Description | | ------------------------- | ------------------------------- | ---------- | --------------------------------- | | shards | ShardConfig[] | Required | Array of shard configurations | | strategy | 'modulo' \| 'consistent-hash' | 'modulo' | Routing algorithm | | createClient | (url, shardId) => TClient | Required | Factory to create Prisma clients | | healthCheckIntervalMs | number | 30000 | Health check frequency | | circuitBreakerThreshold | number | 3 | Failures before marking unhealthy |

Shard Config

interface ShardConfig {
  id: string; // Unique identifier (e.g., 'shard_1')
  url: string; // PostgreSQL connection string
  weight?: number; // Optional weight for distribution
  isReadReplica?: boolean;
}

Routing Strategies

Modulo (Default)

Simple and fast. Uses hash(key) % shardCount for routing.

strategy: 'modulo';

Consistent Hash

Minimizes data movement when adding/removing shards.

strategy: 'consistent-hash';

Error Handling

import { ShardingError, ConfigError, ConnectionError } from 'prisma-sharding';

try {
  const client = sharding.getShard(userId);
} catch (error) {
  if (error instanceof ConnectionError) {
    console.error(`Shard ${error.shardId} unavailable`);
  }
}

Custom Logger

const sharding = new PrismaSharding({
  // ...config,
  logger: {
    info: (msg) => myLogger.info(msg),
    warn: (msg) => myLogger.warn(msg),
    error: (msg) => myLogger.error(msg),
  },
});

getAllClients()

Get all Prisma client instances.

const clients = sharding.getAllClients();

console.log(`Managing ${clients.length} shard clients`);

Returns: PrismaClient[]

getShardCount()

Get total number of configured shards.

const count = sharding.getShardCount();
console.log(`Running on ${count} shards`);
// Output: Running on 3 shards

getShardIds()

Get array of all shard IDs.

const shardIds = sharding.getShardIds();
console.log(shardIds);
// Output: ['shard_1', 'shard_2', 'shard_3']

Returns: string[]

Author

safdar-azeem

License

MIT