@sygnl/dedupe

In-memory event deduplication with time-window tracking for edge workers and high-throughput systems.

Install

npm install @sygnl/dedupe

Quick Start

import { DedupeManager } from '@sygnl/dedupe';

const dedupe = new DedupeManager({
  configs: {
    purchase: {
      enabled: true,
      dedupeKey: 'order.id',
      windowMs: 300000 // 5 minutes
    },
    add_to_cart: {
      enabled: true,
      dedupeKey: 'cart.id',
      windowMs: 60000 // 1 minute
    }
  }
});

// Check for duplicates
const result = dedupe.check('purchase', event, 'evt_123', 'webhook');

if (result.isDuplicate) {
  console.log('Duplicate detected!');
  console.log('Previous event:', result.previousEventId);
  console.log('Age:', result.ageMs, 'ms');
  return; // Skip processing
}

// Process event...

Use Cases

• Prevent double-clicks - Stop users from submitting forms multiple times
• Webhook retries - Avoid processing the same webhook twice
• Multiple sources - Dedupe events from pixel + webhook for same order
• Edge workers - No external storage needed (KV, Redis)
• High throughput - In-memory, sub-millisecond lookups

How It Works

1. Extract key from event (e.g., order.id, user.email)
2. Check cache if key seen within time window
3. Return duplicate if found, or record as new
4. Auto-cleanup expired entries to prevent memory leaks

API

DedupeManager

const dedupe = new DedupeManager({
  configs?: Record<string, DedupeConfig>,
  defaultWindowMs?: number,      // Default: 60000 (1 min)
  cleanupIntervalMs?: number,    // Default: 60000 (1 min)
  maxAgeMs?: number,             // Default: 600000 (10 min)
  debug?: boolean                // Default: false
});

check(eventType, event, eventId, source?, metadata?): DedupeResult

Check if event is a duplicate.

const result = dedupe.check('purchase', {
  order: { id: 'order_123' },
  total: 99.99
}, 'evt_abc', 'webhook');

// Result
{
  isDuplicate: false,
  dedupeKey: 'order_123',
  windowMs: 300000
}

configure(eventType, config): void

Configure dedupe for an event type at runtime.

dedupe.configure('subscription_created', {
  enabled: true,
  dedupeKey: 'subscription.id',
  windowMs: 120000
});

Dedupe Key Extraction

Object paths

// Nested object
{ dedupeKey: 'user.profile.email' }
// Event: { user: { profile: { email: '[email protected]' } } }
// Key: '[email protected]'

Array indexing

// First product ID
{ dedupeKey: 'products.0.id' }
// Event: { products: [{ id: 'prod_123' }, { id: 'prod_456' }] }
// Key: 'prod_123'

Custom extractor

{
  dedupeKey: (event) => {
    // Custom logic
    return event.items.map(i => i.id).join(',');
  }
}
// Event: { items: [{ id: 'A' }, { id: 'B' }] }
// Key: 'A,B'

Examples

E-commerce

const dedupe = new DedupeManager({
  configs: {
    purchase: {
      enabled: true,
      dedupeKey: 'order.id',
      windowMs: 300000 // 5 min - prevent double submissions
    },
    add_to_cart: {
      enabled: true,
      dedupeKey: (e) => `${e.user.id}:${e.product.id}`,
      windowMs: 60000 // 1 min - same user adding same product
    }
  }
});

Webhook Processing

// Cloudflare Worker
export default {
  async fetch(request, env) {
    const event = await request.json();
    
    const result = dedupe.check(
      event.type,
      event,
      event.id,
      'webhook'
    );
    
    if (result.isDuplicate) {
      return new Response('Duplicate', { status: 200 });
    }
    
    await processEvent(event);
    return new Response('OK');
  }
};

SaaS Events

const dedupe = new DedupeManager({
  configs: {
    user_signup: {
      enabled: true,
      dedupeKey: 'user.email',
      windowMs: 120000 // 2 min
    },
    subscription_started: {
      enabled: true,
      dedupeKey: 'subscription.id',
      windowMs: 600000 // 10 min
    }
  }
});

Utility Methods

// Clear all state (testing)
dedupe.clear();

// Get count of tracked events
const count = dedupe.getCount(); // 42

// Get keys for event type
const keys = dedupe.getKeysForType('purchase'); // ['order_123', 'order_456']

// Get specific entry
const entry = dedupe.getEntry('purchase', 'order_123');
// { eventId: 'evt_abc', timestamp: 1234567890, source: 'webhook' }

// Force cleanup
const removed = dedupe.forceCleanup(); // 15

Configuration Options

Per-Event Config

interface DedupeConfig {
  enabled: boolean;
  dedupeKey: string | ((event: any) => string | null);
  windowMs: number;
}

Global Options

interface DedupeManagerOptions {
  configs?: Record<string, DedupeConfig>;
  defaultWindowMs?: number;       // Default window for unconfigured events
  cleanupIntervalMs?: number;     // How often to cleanup expired entries
  maxAgeMs?: number;              // Max age before entry is removed
  debug?: boolean;                // Enable debug logging
}

Time Windows

Common window durations:

• 1 second: 1000
• 30 seconds: 30000
• 1 minute: 60000
• 5 minutes: 300000
• 10 minutes: 600000
• 1 hour: 3600000

Performance

• Lookup: O(1) - Hash map lookup
• Cleanup: O(n) - Runs every cleanupIntervalMs
• Memory: ~200 bytes per tracked event

Edge Cases

• Missing keys: Pass through (not duplicate)
• Null events: Pass through
• No config: Pass through
• Disabled: Pass through
• Expired: Treated as new event

License

Apache-2.0