Raptiye (Raptiye Consensus & Replicated Log Core)

Raptiye is a high-performance, byte-first replicated-log and consensus core for JavaScript and Node.js with zero runtime dependencies.

Raptiye is designed as a reusable low-level primitive for distributed databases, key-value stores, caches, queues, event logs, metadata services, and replicated state machines.

Core Principles

1. Byte-First Design: Application commands are opaque Uint8Array binary payloads. Raptiye does not decode or inspect application mutations.
2. Zero Runtime Dependencies: Built entirely using Node.js built-ins (node:net, node:fs, node:crypto, node:test, node:assert).
3. Zero-Copy Scatter/Gather Protocol: Message framing uses scatter/gather arrays (sendv) allowing headers, metadata, and application payload slices to be transmitted without concatenating or copying bytes.
4. Pure Deterministic State Machine: The consensus core (ConsensusEngine.step(event) => effects) has zero side effects, enabling discrete-event simulation, randomized chaos testing, and reproducible seed testing without sockets or real timers.
5. Raft Consensus Safety:
   • Four node roles: FOLLOWER, PRE_CANDIDATE, CANDIDATE, LEADER.
   • Pre-Vote implemented from the start to prevent disruptive term bumps from isolated nodes.
   • Quorum commits (e.g. 2 of 3) with monotonic commit tracking.
   • Split-brain resistance: isolated leaders cannot commit and automatically step down upon partition heal.
   • Pipelined replication with bounded inflight windows (maxInflightBatches, maxInflightBytes) and slow follower isolation.
   • Log compaction & snapshot streaming.
   • Controlled zero-disruption leadership transfer (TIMEOUT_NOW).
   • Crash-safe write-ahead log (FileLog) with atomic fsync barriers.

Performance Dashboard

Measured on Node.js v20 (Apple M2, Darwin arm64):

| Benchmark Metric | Measured Result | Architectural Assessment | | :--- | :--- | :--- | | Single-Node Overhead | 2,443,326 ops/sec (409 ns/op, 160 B/op) | Ultra Low Core Overhead | | 3-Node Replication (64B) | 63,449 cmd/sec (3.87 MB/s) | Fast Quorum Throughput | | Commit Latency p50 (64B) | 0.012 ms (12 µs) | Sub-Millisecond Quorum | | Commit Latency p99 (64B) | 0.046 ms (46 µs) | Deterministic Latency | | Copied Bytes / Byte | 0.00 | True Zero-Copy Scatter/Gather | | Headline Failover (T7-T0 p50)| 133 ms | Fast Automatic Failover | | Headline Failover (T7-T0 p99)| 208 ms | Predictable Recovery | | Slow Follower Isolation | 333 ops/sec (Slow RTT 200ms) | Fast Quorum Unblocked | | Follower Catch-up (100K) | 1,533,023 entries/sec (65.23 ms) | High-Speed Batched Recovery | | Event-Loop Lag p99 | 0.024 ms (24 µs) | Responsive Node.js Runtime | | Memory Soak (50K Ops) | Plateau at 77 MB | Stable & Bounded Memory |

Architecture

                    RAPTIYE NODE
                         │
    ┌────────────────────┼────────────────────┐
    ▼                    ▼                    ▼
Consensus Engine     Replication Pipeline    Log Storage
(Pure State Machine) (Scatter/Gather sendv)  (MemoryLog / FileLog)
    │                    │                    │
    ├─ Pre-Vote          ├─ 28B Header        ├─ Append-Only WAL
    ├─ Raft Election     ├─ Inflight Windows  ├─ Atomic Hard State
    ├─ Quorum Commit     ├─ Backpressure      ├─ Log Compaction
    ├─ Split-Brain Guard └─ TCP / Memory Net  └─ Snapshots
    └─ Step FSM

Binary Wire Protocol Layout

Each message begins with a fixed 28-byte common header:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Magic (0x5250)       |  Ver (0x01)   |  MessageType  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|             Flags             |             Term              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               +
|                      Term (64-bit uint)                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          SourceNode           |        DestinationNode        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         PayloadLength                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        CRC32 Checksum                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Reserved            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

When transmitting APPEND_REQUEST, the encoder creates scatter/gather chunks: [HeaderBuffer (28B), MetadataBuffer, ...payloadBuffers] The payload buffers are passed directly to sendv without copying or concatenation.

Usage Example

import { Raptiye, MemoryNetwork, MemoryTransport, MemoryLog } from 'raptiye';

const net = new MemoryNetwork();

const node1 = new Raptiye({
  id: 1,
  peers: [2, 3],
  storage: new MemoryLog(),
  transport: new MemoryTransport(1, net),
  apply: (entry) => {
    console.log(`Applied log index ${entry.index}:`, entry.payload);
  }
});

const node2 = new Raptiye({
  id: 2,
  peers: [1, 3],
  storage: new MemoryLog(),
  transport: new MemoryTransport(2, net)
});

const node3 = new Raptiye({
  id: 3,
  peers: [1, 2],
  storage: new MemoryLog(),
  transport: new MemoryTransport(3, net)
});

// Start nodes
await Promise.all([node1.start(), node2.start(), node3.start()]);

// Submit an opaque binary payload to leader
const payload = new Uint8Array([0xCA, 0xFE, 0xBA, 0xBE]);
const index = node1.submit(payload);

// Await consensus quorum commit
await node1.committed(index);

console.log('Leader stats:', node1.stats());

// Graceful leadership transfer
await node1.transferLeadership(2);

// Shutdown
await Promise.all([node1.shutdown(), node2.shutdown(), node3.shutdown()]);

Running Tests & Benchmarks

Test Suite

Runs all 21 unit, integration, and randomized chaos tests:

npm test

Benchmark Suite

Runs all 9 benchmarks and generates bench-results.json:

npm run bench

Individual benchmarks:

• node bench/bench-single-node.js: Core submit overhead (ns/op, ops/sec)
• node bench/bench-replication.js: 3-node replication across payload sizes (16B to 1MB)
• node bench/bench-batch-curve.js: Batch size impact (1 to 4096 entries)
• node bench/bench-pipeline.js: Inflight replication pipeline depth
• node bench/bench-slow-follower.js: Quorum throughput under 200ms slow follower
• node bench/bench-failover.js: Detailed T0..T7 failover timeline
• node bench/bench-partition.js: Split-brain resistance & partition healing
• node bench/bench-recovery.js: Follower catch-up throughput (1K, 100K entries)
• node bench/bench-event-loop.js: Event-loop lag under load
• node bench/bench-memory.js: Sustained memory soak test with log compaction