Daktilo

Daktilo is a high-performance, durable, ordered binary append-only log engine for Node.js, designed to serve as foundational infrastructure for modern databases (such as Relbyte and Tencere).

It is NOT a database, query engine, replication engine, or transaction engine.

Its responsibility is simple and uncompromising:

Accept binary records, assign them an ordered sequence number, persist them safely and efficiently, and allow them to be replayed, streamed, replicated, checkpointed, and recovered after crashes.

Architecture & Split of Responsibilities

cemalloc  ────────►  memory / zero-copy allocation
Muttafa   ────────►  storage data structures
Daktilo   ────────►  durable ordered history (WAL & Commit Log)
Raptiye   ────────►  transport / distribution & replication
Relbyte   ────────►  relational database semantics
Tencere   ────────►  key-value / cache / coordination semantics

Core Principles

• Append-only binary log: Zero JSON overhead; records are framed binary buffers.
• Monotonically increasing sequence numbers: Sequence numbers are strictly ordered bigints.
• Single-writer architecture: Internal MPSC queue guarantees deterministic write ordering without lock contention.
• Adaptive batching & Group Commit: High concurrency automatically coalesces writes into single fdatasync barriers.
• In-Memory Hot-Read Ring Cache: 124x faster sub-microsecond RAM reads for recent commits and streaming replication.
• Configurable durability: strict, batch (default), or async.
• Integrity protection: High-throughput slicing-by-8 Castagnoli CRC32C checksums protect all headers, records, and blocks.
• Deterministic crash recovery: Active segment crash-tail truncation recovers clean state; historical sealed segments throw structured corruption errors on bit-rot.
• Sparse indexing: In-memory binary-searchable typed sparse index enables fast seeks with near-zero memory footprint.
• Checkpoints & Compaction: Safe log compaction reclaims historical segments strictly below the application checkpoint.
• Live tailing: Bounded async iterable streaming with backpressure for CDC and replication.
• Worker Thread block-level compression: Background Zstandard block compression offloaded to worker threads with adaptive compressibility fallbacks.

Installation

npm install daktilo

Quick Start

import { Daktilo } from "daktilo";

// Open log storage
const log = await Daktilo.open("./data", {
  durability: "batch" // 'strict' | 'batch' (default) | 'async'
});

// Append binary payload
const seq = await log.append(Buffer.from("hello world"));
console.log(`Appended record at sequence: ${seq}`);

// Append batch of records atomically
const seqs = await log.appendMany([
  Buffer.from("event-1"),
  Buffer.from("event-2")
]);

// Read individual record by sequence
const entry = await log.read(seq);
console.log(`Seq: ${entry.sequence}, Payload: ${Buffer.from(entry.payload).toString()}`);

// Sequential range scan
for await (const entry of log.range(1n, 100n)) {
  console.log(`Replayed: ${entry.sequence}`);
}

// Live tailing (CDC / Streaming)
for await (const entry of log.tail({ from: 1n })) {
  console.log(`Live: ${entry.sequence}`);
  break;
}

// Checkpoint and compact obsolete segments
await log.checkpoint(100n);
await log.compact();

// Graceful close
await log.close();

Durability Semantics

Daktilo never obscures durability mode:

| Mode | Filesystem Barrier | Resolution | Guarantees | | :--- | :--- | :--- | :--- | | batch (Default) | Coalesced fdatasync | Resolves after shared sync | Zero data loss on crash. Concurrent appends share sync barrier. | | strict | Dedicated fdatasync | Resolves after dedicated sync | Zero data loss on crash. Dedicated sync barrier per append. | | async | Periodic flush | Resolves immediately on OS write | Maximum ingestion throughput. Acknowledged writes may be lost on power loss prior to flush(). |

See docs/DURABILITY.md for full formal specifications.

Segment & Binary Record Format

Every record is framed with an 18-byte little-endian header:

┌────────────┬────────────┬────────────┬────────────┬───────────────┐
│ length     │ sequence   │ flags      │ checksum   │ payload       │
│ uint32 (4B)│ uint64 (8B)│ uint16 (2B)│ uint32 (4B)│ N bytes       │
└────────────┴────────────┴────────────┴────────────┴───────────────┘

Segments begin with a fixed 64-byte header (DFT1 magic, version, flags, segment ID, first sequence, and CRC32C checksum).

See docs/BINARY_FORMAT.md for full binary layout.

Background Block-Level Compression

Compression is completely decoupled from the hot append path:

1. Active segment appends are always raw: Zero CPU or compression latency on writes.
2. Background worker: Sealed segments are compressed in the background into independent blocks (default: 256 KiB).
3. Adaptive compression: Incompressible data automatically bypasses Zstandard and is stored as RAW blocks.
4. Fast random reads: Random reads decompress only the single targeted block (< 0.5 ms), avoiding full-segment decompression.

See docs/COMPRESSION.md for details.

Crash Recovery & Torture Testing

Daktilo guarantees deterministic crash recovery:

• Active crash-tail: Incomplete headers or torn writes at the active segment EOF are truncated to the last completely valid record.
• Sealed segment corruption: Bit-rot in historical sealed segments throws structured CorruptionError.
• Torture Test: Tested under repeated, random hard SIGKILL (kill -9) termination in test/torture.test.js, proving 100% data preservation of acknowledged writes.

See docs/RECOVERY.md.

CLI Tooling

Daktilo ships with built-in inspection and verification binaries:

Inspect Storage

npx daktilo inspect ./data

Displays format version, manifest generation, active segment, checkpoint sequence, head sequence, and segments table.

Deep Integrity Verification

npx daktilo verify ./data

Scans all segment headers, record framing, sequence monotonicity, and CRC32C checksums.

Dump Log Records

npx daktilo dump ./data --from 1000 --to 1050
npx daktilo dump ./data --hex

Benchmarks

Key results on Apple Silicon (macOS, NVMe):

• Async Append: 44,980 ops/sec (6.26 MB/sec, p50: 0.018 ms)
• Batch AppendMany: 29,588 ops/sec (4.12 MB/sec)
• Concurrency Scaling (Batch Durability):
  • 1 Producer: 441 ops/sec
  • 16 Producers: 5,064 ops/sec
  • 256 Producers: 83,273 ops/sec (11.59 MB/sec under durable fsync!)
• Bandwidth: 136.55 MB/sec (1 MiB payloads)
• Crash Recovery Scan: ~32,000 records/sec
• Block Random Read (Zstd): 0.196 ms per lookup

See bench/BENCHMARK_REPORT.md for full benchmarks and latency percentiles.

Custom Database WAL Example

See examples/database-wal.js for an end-to-end example demonstrating how an external database encodes custom operations (PUT, DELETE), appends to Daktilo, checkpoints state, and replays history on startup.

License

Apache-2.0