Teyyare

Teyyare is a high-performance, resumable file transfer engine and CLI built directly on top of:

• muttafa: low-level mutation buffering, generation freeze & lifecycle engine
• raptiye: high-performance byte-first replicated log & TCP binary transport

Teyyare is the first real integration workload for both projects, proving that a mutation lifecycle engine and a binary replication/transport engine compose into a fast, bounded-memory, resumable file transfer system without introducing redundant batching layers or memory copies.

                    TEYYARE
                       │
               File / Directory API
                       │
                       ▼
                    MUTTAFA
              transfer write pipeline
                       │
                    freeze
                       │
                       ▼
              Frozen Generation
                       │
                       ▼
                    RAPTIYE
             replication / transport
                       │
                       ▼
                      TCP (Multi-Channel)
                       │
                       ▼
                 Remote Teyyare
                       │
                       ▼
                    MUTTAFA
                       │
                     flush
                       │
                       ▼
                     Disk (writev + pre-allocation)

Core Responsibility Split

| Layer | Responsibility | |---|---| | Muttafa | Chunk ingestion, mutation buffering, generation lifecycle, freeze, immutable batches, backpressure, memory accounting. | | Raptiye | Peer connection, TCP transport, binary framing, replication stream, ACKs/progress, delivery ordering, flow control. | | Teyyare | File/directory semantics, manifest, chunks, checksums, resume state, file reconstruction, verification, atomic finalization (.teyyare-part -> final), CLI. |

Architecture & Systems Optimizations

1. Sender Hot Path

FileHandle.read()
      │
      ▼
chunk buffer
      │
      ▼
crc32 checksum
      │
      ▼
Muttafa mutation (OP_CHUNK)
      │
      ▼
MutableGeneration
      │
    freeze
      │
      ▼
FrozenGeneration [metaBuf, payloadArena]
      │
      ▼
Raptiye.submitBatch() (In-Flight Sliding Window up to 128 MB)
      │
      ▼
Multi-Channel TCP Pool (Zero-Allocation Round-Robin sendv)

2. Receiver Hot Path

Multi-Channel TCP
 │
 ▼
Zero-Copy Pointer Framer (O(1) cursor, no shift(), scratch header reuse)
 │
 ▼
Raptiye (Frame decode + verification)
 │
 ▼
FrozenGeneration (O(1) zero-copy view via FrozenArenaStore.fromBuffers)
 │
 ▼
Chunk Coalescing Engine (Merges contiguous chunk runs)
 │
 ▼
FileHandle.writev(buffers, startOffset) (Direct native scatter-gather writes)
 │
 ▼
Debounced Asynchronous State Persistence (.teyyare-state)

3. Key Systems Optimizations Implemented

• Disk Pre-Allocation (FileHandle.truncate(targetSize)): Zero-fragmentation disk pre-allocation upon receiving FILE_BEGIN, eliminating filesystem dynamic block allocation latency during high-speed writes.
• Native Scatter-Gather Writes (writev): Receiver coalesces contiguous chunks arriving in the same generation batch into a single system call via writev, reducing write syscalls by ~87.5%.
• Debounced Asynchronous State Saving: .teyyare-state persistence is debounced to disk (every 250ms or 32 chunks) and drained cleanly before final file rename, completely eliminating blocking JSON serialization from the hot-path.
• In-Flight ACK Pipelining (Sliding Window): Supports up to 32 concurrent batches and 128 MB in-flight capacity. Senders stream generations continuously across 4–8 parallel TCP channels without stalling on single-batch ACKs.
• Pointer-Based Zero-Copy TCP Framer: Eliminated $O(N)$ array shift() operations and repeated Buffer.allocUnsafe calls on the TCP stream reader using an $O(1)$ cursor and amortized array compaction.
• Zero-Allocation Channel Selection: Eliminated pool.filter garbage collector churn in TCPTransport.prototype.sendv and send using an in-place circular lookup.
• Dynamic WAN Tuning: PULL_REQUEST carries client-specified --chunk-size (e.g. 2 MB) and --batch-size (e.g. 16 MB) to saturate high-latency, high-bandwidth WAN connections.
• Multi-Channel Out-of-Order Packet Resilience: Resilient state reconciliation for TRANSFER_BEGIN, FILE_BEGIN, FILE_END, and TRANSFER_END packets arriving out-of-order across parallel TCP sockets.

CLI Usage

Start Receiver Server

# Listen on default port 7421
teyyare serve --port 7421 --dir /destination/folder

Send File or Directory (Push)

# Single file transfer with 4 parallel channels
teyyare send ./large-file.bin 127.0.0.1:7421:/tmp/large-file.bin -c 4

# Directory transfer with nested hierarchy
teyyare send ./dist server.local:7421:/var/www/app -c 4

Pull File or Directory (Download)

# Single file download from remote server with 8 channels and 2MB WAN chunks
teyyare pull 38.242.216.102:7421:/root/cdn/app.zip ./ -c 8 --chunk-size 2M --batch-size 16M

# Directory pull
teyyare pull server:7421:/backup/db.tar.zst ./ -c 4

CLI Options

| Flag | Description | Default | |---|---|---| | -c, --channels <n> | Number of parallel TCP socket channels | 4 (remote), 1 (local) | | --chunk-size <size> | Chunk size, e.g. 512K, 1M, 2M, 4M | 2M (remote), 1M (local) | | --batch-size <size> | Generation batch size, e.g. 8M, 16M | 8x chunk size | | --port <port> | Server port number | 7421 | | --dir <path> | Root directory for serve command | Current directory (./) |

Progress Display

Teyyare [Download]

app.zip

333.54 MB / 333.54 MB
100.0%

network speed    324.5 MB/s
disk speed       1240.2 MB/s
channels         8
ETA              --:--

chunks           167 / 167
generation       21
verified CRC32   167
resume hits      0

Test Suite & Softscope Profiling

1. Unit & Integration Tests

Run all 24 automated tests covering protocol encoding, generation ownership, backpressure, chaos recovery, and multi-channel transfers:

npm test

2. End-to-End (E2E) Test Suite

Runs 4 comprehensive end-to-end scenarios (Single Upload, Directory Upload, Single Pull, 50% Resumed Transfer):

npm run e2e

3. Softscope Profiling

Profile CPU bottlenecks, hot call paths, and memory allocations using softscope:

# Markdown summary to stdout
npm run e2e:md

# Full interactive terminal dashboard (TUI)
npm run e2e:profile

# Live metrics during execution
npm run e2e:live

Benchmark & Throughput Results

End-to-End Suite Summary (test/e2e/e2e.js)

| Scenario | Transfer Size | Duration | Throughput | Verification | |---|---|---|---|---| | Single File Upload (4 channels) | 32.0 MB | 121ms | 264.5 MB/s | SHA-256 Match | | Directory Upload (Nested tree) | 16.0 MB | 104ms | 153.9 MB/s | All files verified | | Single File Pull (4 channels) | 32.0 MB | 89ms | 359.6 MB/s | SHA-256 Match | | Resumed Transfer (50% skipped) | 8.0 MB | 49ms | 163.3 MB/s | 0 retransmitted bytes |