🔁 idempotency-tester

Fire the same request N times. Check what actually landed in the database.

Language: English | 한국어

Contents

• 😬 The problem
• ✨ Features
• 📦 Installation
• 🚀 Quick start
• 🧩 Scenario spec
• ⚙️ Configuration
• 🖥️ CLI reference
• 🤖 GitHub Action
• 🛠️ Programmatic API
• 🔌 Supported protocols
• 🤝 Contributing
• 📄 License

😬 The problem

Your API returns 200 OK. Your Kafka consumer says it processed the event. Your tests are green.

Then a client retries a timed-out request, or a broker redelivers a message after a rebalance — and suddenly there are two charges, two shipments, or two rows where there should be one.

Idempotency bugs don't show up in normal tests. They only show up when the same request/event arrives more than once, often at the same instant. idempotency-tester exists to force that moment on purpose:

   ┌────────────────┐
   │ scenario        │
   │ (*.yaml)        │
   └───────┬─────────┘
           │  send the same payload
           │  100×, concurrently
           ▼
   ┌────────────────┐
   │ your API /      │
   │ consumer        │
   └───────┬─────────┘
           │  wait, then check
           ▼
   ┌────────────────┐
   │ SELECT COUNT(*) │
   │ ... WHERE key=? │
   └───────┬─────────┘
           │
           │  expected: 1
           │  actual:   ?
           ▼
   ✅ PASS  ❌ FAIL  🚧 BLOCKED

No mocks. No stubbing the handler. It hits the real target and reads the real table.

✨ Features

• 🎯 Deterministic verdicts — PASS / FAIL / BLOCKED, decided by comparing an expected row count against what's actually in the database. No guessing from HTTP status codes.
• ⚡ Three concurrency modes — parallel (true race condition), burst (jittered retry storm), staggered (sequential redelivery) — because "duplicate" means different things on different protocols.
• 🔌 Pluggable protocols — HTTP, Kafka, SQS, RabbitMQ out of the box, plus a custom protocol adapter API for gRPC/GraphQL/anything else. Each adapter's dependency (kafkajs, amqplib, @aws-sdk/client-sqs, pg/mysql2) is optional — install only what you use.
• 🧠 Built-in diagnosis — a FAIL isn't just a number mismatch; the tool tells you what kind of duplication pattern it saw (no key check at all vs. a partial race condition).
• 🎲 Flaky race detection — --repeat N reruns each scenario N times and fails the whole thing if even one run reveals a duplicate, so a race condition that only fires 1-in-5 doesn't slip through as a lucky PASS.
• 📄 CI-friendly — non-zero exit code on any FAIL, a JSON results file, a Markdown report, and a GitHub Action that annotates the PR directly.
• 🪶 Small footprint — two runtime dependencies (commander, js-yaml). Uses the Node 18+ built-in fetch for HTTP, no axios required.

📦 Installation

npm install --save-dev idempotency-tester

Then install the client library for whichever protocol(s) and database you're testing against:

# pick what you need
npm install --save-dev kafkajs            # protocol: kafka
npm install --save-dev amqplib            # protocol: rabbitmq
npm install --save-dev @aws-sdk/client-sqs # protocol: sqs
npm install --save-dev pg                 # db: postgres
npm install --save-dev mysql2             # db: mysql

(HTTP needs nothing extra — it uses the native fetch.)

🚀 Quick start

npx idempotency-tester init

This scaffolds:

idempotency.config.yaml     # connection details (brokers, base URL, DB)
scenarios/
  ├── kafka-duplicate-key.yaml
  ├── http-idempotency-key.yaml
  ├── sqs-visibility-timeout.yaml
  └── rabbitmq-redelivery.yaml

Edit idempotency.config.yaml with your real connection info, edit the scenario(s) for the protocol you actually use (delete the rest), then:

npx idempotency-tester run

Running http-payment-idempotency-key-100x (http, 100x parallel)...
  -> FAIL

[FAIL] http-payment-idempotency-key-100x
        3 records were persisted instead of the expected 1 — idempotency checking is only
        partially effective (e.g. a race condition around a check-then-insert, a missing
        unique constraint, or a TTL-limited dedup cache).

Summary: 0 passed, 1 failed, 0 blocked
Report: reports/idempotency-report-20260827.md

Exit code is 1 whenever any scenario FAILs — drop it straight into CI.

🧩 Scenario spec

Each scenarios/*.yaml file describes one duplication scenario:

scenario_id: kafka-duplicate-key-100x
protocol: kafka                    # kafka | http | sqs | rabbitmq | custom
target:
  description: Order-created Kafka consumer
  topic: orders.created            # http: "METHOD /path", sqs: queue URL, rabbitmq: queue name
idempotency_key:
  field: orderId
  value: test-order-001            # same value on every fire — that's the "duplicate"
concurrency:
  count: 100
  mode: parallel                   # parallel | burst | staggered
payload_template:
  orderId: "{{idempotency_key}}"   # substituted with idempotency_key.value
  amount: 1000
settle_wait_ms: 2000               # optional, defaults to 2000ms — async consumer catch-up time
expected_behavior:
  db_check:
    table: orders
    query: "SELECT COUNT(*) FROM orders WHERE order_id = :key"
    expected_count: 1
notes: >
  Simulates at-least-once redelivery after a consumer group rebalance.

| Field | Required | Notes | |---|---|---| | scenario_id | ✅ | Unique identifier, used in output and file names | | protocol | ✅ | kafka | http | sqs | rabbitmq | custom | | target.topic | ✅ | Protocol-specific address (see table above) | | target.module / target.export | – (required for custom) | Path to a custom protocol adapter module + export name (defaults to send) | | idempotency_key.value | ✅ | Kept constant across all fires — this is the duplication | | concurrency.count / mode | ✅ | How many fires, and how they're timed | | payload_template | – | {{idempotency_key}} is replaced with the key value anywhere in the object | | settle_wait_ms | – | Wait before checking the DB (async consumers need catch-up time) | | expected_behavior.db_check.expected_count | ✅ | The number PASS/FAIL is judged against | | expected_behavior.db_check.query | – (required for postgres/mysql) | Must return a single count column. Omit entirely when using a custom verifier |

Concurrency modes

| Mode | Timing | Reveals | |---|---|---| | parallel | Promise.all, all fires start together | The strongest race condition | | burst | Each fire waits 0–50ms (random) before sending | A realistic client/broker retry storm | | staggered | Fixed 10ms gap, sent one at a time | Whether even non-racing duplicates are still deduplicated |

⚙️ Configuration

idempotency.config.yaml holds connection details shared by all scenarios:

connections:
  kafka:
    brokers: [localhost:9092]
  http:
    base_url: http://localhost:3000
    idempotency_header: Idempotency-Key
  sqs:
    region: us-east-1
    fifo: false
  rabbitmq:
    url: amqp://localhost
  db:
    type: postgres              # postgres | mysql
    connection_env: DATABASE_URL   # reads process.env.DATABASE_URL — keeps secrets out of the file

You only need to fill in the block(s) for the protocol(s) your scenarios actually use.

🧬 Custom verifiers (MongoDB, Redis, anything)

db.type isn't limited to postgres/mysql — set it to custom and point at your own module. This is how you verify against MongoDB, Redis, DynamoDB, or even an internal admin API, without idempotency-tester needing a driver for it:

connections:
  db:
    type: custom
    module: ./verifiers/mongo-verifier.js   # resolved relative to this config file
    export: verify                          # optional, defaults to "verify"

// verifiers/mongo-verifier.js
import { MongoClient } from "mongodb";

const client = await new MongoClient(process.env.MONGO_URL).connect();
const orders = client.db("shop").collection("orders");

export async function verify({ keyValue }) {
  return orders.countDocuments({ orderId: keyValue });
}

export async function close() {
  await client.close(); // optional — called once after all scenarios finish
}

With a custom verifier, expected_behavior.db_check.query in your scenario file becomes optional — only expected_count is required. The same escape hatch is available from the programmatic API via run({ ..., customVerifier }), which takes precedence over the config file entirely. See Programmatic API below.

🔌 Custom protocol adapters (gRPC, GraphQL, anything)

protocol isn't limited to kafka/http/sqs/rabbitmq — set it to custom and point target at your own module. This is how you fire duplicates over gRPC, GraphQL, WebSocket, or an internal RPC framework without idempotency-tester needing a built-in adapter for it:

scenario_id: grpc-duplicate-payment
protocol: custom
target:
  description: gRPC PaymentService.CreatePayment
  topic: grpc://payments/CreatePayment   # free-form label — not parsed, just shown in output
  module: ./adapters/grpc-adapter.js     # resolved relative to idempotency.config.yaml, same as a custom verifier's module
  export: send                           # optional, defaults to "send"
idempotency_key:
  field: orderId
  value: test-order-001
concurrency:
  count: 100
  mode: parallel
payload_template:
  orderId: "{{idempotency_key}}"
expected_behavior:
  db_check:
    table: payments
    query: "SELECT COUNT(*) FROM payments WHERE order_id = :key"
    expected_count: 1

// adapters/grpc-adapter.js
import { credentials } from "@grpc/grpc-js";
import { PaymentServiceClient } from "./generated/payment_grpc_pb.js";

const client = new PaymentServiceClient("payments.internal:50051", credentials.createInsecure());

export async function send(target, payload) {
  const start = Date.now();
  try {
    await new Promise((resolve, reject) =>
      client.createPayment(payload, (err) => (err ? reject(err) : resolve()))
    );
    return { succeeded: true, latencyMs: Date.now() - start };
  } catch (err) {
    return { succeeded: false, latencyMs: Date.now() - start, error: String(err) };
  }
}

export async function close() {
  client.close(); // optional — called once after all scenarios finish
}

The exported function's signature ((target, payload) => Promise<{ succeeded, latencyMs, error? }>) is exactly ProtocolAdapter.fire — a custom adapter module is an adapter, just loaded at runtime instead of built in. Different scenarios can point at different modules (or the same one), each cached and connected once per run.

🖥️ CLI reference

idempotency-tester init                        # scaffold config + example scenarios
idempotency-tester run [scenarioPath]           # default: ./scenarios
  -c, --config <path>   connections config       (default: idempotency.config.yaml)
  -o, --out <dir>       output directory         (default: reports)
  -r, --repeat <n>      repeat each scenario N times, FAIL on any failing run (default: 1)
  -w, --watch           re-run on scenario/config file changes

idempotency-tester annotate                     # for CI — see GitHub Action below
  -r, --results <path>  path to results.json     (default: reports/results.json)
  --comment              upsert a summary comment on the pull request
  --token <token>        GitHub token for the PR comment (default: GITHUB_TOKEN/GH_TOKEN env var)

idempotency-tester diff <baselinePath>           # compare against a baseline results.json
  -c, --current <path>  path to the current run's results.json (default: reports/results.json)

--watch — re-runs the whole scenario set (debounced) for local development:

• Triggers on any change under scenarioPath, the config file, or a referenced custom verifier/custom protocol adapter module
• Always re-imports the latest saved version of a changed module, never a stale cached copy
• A development convenience only — not meant for CI
• A custom module referenced by a scenario added after --watch starts isn't picked up until the CLI is restarted

--repeat — reruns each scenario to catch race conditions that don't reproduce on every run:

• A parallel mode with 100 requests can pass once by luck and still have a real bug underneath
• --repeat 5 runs a scenario 5 times sequentially — never concurrently with itself, since all 5 share one fixed idempotency key and overlapping runs would corrupt each other's DB state
• The aggregate verdict is FAIL if any one of the runs failed; each attempt's full result is kept under repeat.attempts in results.json so you can see exactly which run(s) failed
• The default --repeat 1 produces output identical to versions before this flag existed — no repeat field appears

diff — compares a results.json against a baseline by scenarioId, so a PR fails only on new violations:

• Regression = a scenario going PASS/BLOCKED → FAIL, or a brand-new scenario failing on its first run
• FAIL → PASS is reported as fixed; a FAIL that was already FAIL in the baseline is reported but doesn't fail the build
• Exit code is 1 when there's at least one regression, 0 otherwise

# on the base branch
idempotency-tester run -o baseline
# on the PR branch
idempotency-tester run
idempotency-tester diff baseline/results.json -c reports/results.json

Every run writes:

• reports/results.json — full structured results, for scripting/CI
• reports/idempotency-report-YYYYMMDD.md — human-readable report, FAILs sorted first

🤖 GitHub Action

Run scenarios in CI and surface violations directly on the pull request — inline ::error:: annotations on the failing scenario file, a step summary, and an upserted PR comment (one comment kept up to date across pushes, not a new one every time):

name: Idempotency

on:
  pull_request:
  push:
    branches: [main]   # populates the baseline cache used by PRs — see `baseline` below

jobs:
  idempotency:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write   # required for the PR comment
      actions: write         # required for the baseline cache
    steps:
      - uses: actions/checkout@v5
      - uses: hhw12409/[email protected]
        with:
          scenario-path: scenarios
          config: idempotency.config.yaml
          repeat: 3           # optional — catch flaky race conditions

The action shells out to npx idempotency-tester, so it needs no build step of its own — bring your own environment (services, DATABASE_URL, etc.) the same way you would for idempotency-tester run locally. It fails the job when any scenario ends up FAIL; BLOCKED scenarios are reported but don't fail the build (see Deterministic verdicts for why).

Baseline-aware gating (baseline: true, the default):

• On a pull_request event: restores a cached results.json from the last run on the PR's base branch, if one exists, and gates the build with diff instead of the raw run — so it fails only on new violations, not ones a previous PR already introduced
• On any other event (a push to your base branch, per the workflow above): saves this run's results.json as the new baseline for that branch
• No push trigger configured → no baseline is ever cached, so this is a no-op and the action falls back to gating on the full run, same as before this input existed
• Check the baseline-used output to see which mode a given run took

| Input | Default | Description | |---|---|---| | scenario-path | scenarios | Scenario file or directory | | config | idempotency.config.yaml | Connections config path | | out | reports | Output directory | | repeat | 1 | Repeat each scenario N times to catch flaky races | | baseline | true | Gate PRs on new violations only, via a cached baseline from the base branch (see above) | | version | latest | npm version/tag of idempotency-tester to run | | working-directory | . | Directory to run in | | comment | true | Upsert a PR comment (no-op outside pull_request/pull_request_target) | | github-token | ${{ github.token }} | Token for reading/posting the PR comment |

Outputs: verdict (pass/fail, reflects only new violations when baseline-used is true), baseline-used (true/false), results-path.

Already have your own workflow and just want the annotations? idempotency-tester annotate -r reports/results.json --comment does the same reporting step standalone, after your own idempotency-tester run call.

🛠️ Programmatic API

import { run } from "idempotency-tester";

const summary = await run({
  scenarioPath: "scenarios",
  configPath: "idempotency.config.yaml",
  outDir: "reports",
});

if (summary.hasFailures) {
  console.error("Idempotency violations found:", summary.results.filter((r) => r.verdict === "FAIL"));
  process.exit(1);
}

Pass a customVerifier to skip the config file's db block entirely and verify with your own function — handy for scripts and test suites that already have a database connection open:

import { run, defineVerifier } from "idempotency-tester";

const verify = defineVerifier(async ({ keyValue }) => {
  return myAlreadyOpenDbConnection.count({ orderId: keyValue });
});

await run({ scenarioPath: "scenarios", configPath: "idempotency.config.yaml", outDir: "reports", customVerifier: verify });

🔌 Supported protocols

| Protocol | Adapter dependency | What "duplicate" simulates | |---|---|---| | http | (none — uses native fetch) | Client retry after timeout, reused Idempotency-Key header | | kafka | kafkajs | At-least-once redelivery after consumer group rebalance | | sqs | @aws-sdk/client-sqs | Message reappearing after visibility timeout expiry | | rabbitmq | amqplib | Broker redelivery after a delayed/failed ack | | custom | (your own module) | Anything else — gRPC, GraphQL, WebSocket, internal RPC. See Custom protocol adapters |

Database verification supports PostgreSQL (pg) and MySQL (mysql2), plus any store via a custom verifier.

🤝 Contributing

1. Fork the repo and create a branch
2. npm install && npm test
3. Open a PR — please include a test for behavior changes

See CONTRIBUTING.md for code conventions and the full pre-PR checklist (npm run lint, npm run format:check, npm run typecheck, npm test).

git clone https://github.com/hhw12409/idempotency-tester.git
cd idempotency-tester
npm install
npm test

📄 License

MIT © hhw12409