buf-im

A zero-allocation byte engine. Bytes are bytes; a question about them costs nothing.

buf-im adds exactly the byte operations JavaScript is missing — and nothing it already has. It never re-implements subarray, set, indexOf(byte) or DataView.getUint32. It implements the things you keep hand-writing (with bugs) or reaching for Node's Buffer to get, and then cannot take to a browser: sub-array search, memcmp, constant-time equality, bit fields at a bit offset, 3-, 6- and 16-byte integers, LEB128 varints, CRC-32, splitting and trimming without copying, byte histograms and entropy, file-type sniffing, a hex dump.

Every method takes any run of bytes — a Uint8Array or Buffer, any other ArrayBufferView, an ArrayBuffer, an Array of byte values — and reads it through one adapter resolved once per call. Nothing is copied on the way in. If you only ask a question ("equal?", "how many distinct bytes?", "is this a PNG?"), nothing is allocated at all.

import b from '@slnknrr/buf-im';

b.eq(png, otherPng);              // 1 — same bytes, same kind; -1 would mean same bytes through another kind of view
b.magic(file);                    // 'png' — from the signature, no parser, no dependency
[...b.split(log)];                // lines as zero-copy views: \r\n, \n and \r are each one boundary
b.uint(hdr, 4, 3);                // a 24-bit big-endian field — DataView stops at 8, 16, 32, 64
b.uvar(frame, 0);                 // 300 — a protobuf / wasm varint, decoded
b.crc32(chunk);                   // 0x414fa339 — the same number zlib.crc32 gives, in a browser too
b.ent(payload);                   // 7.98 — Shannon entropy per byte: compressed or encrypted
b.teq(mac, expected);             // 1 — constant-time, for the comparison that must not leak

What it is

• A precision tool for bytes the standard library fumbles. Cross-platform: everything here works the same on a Uint8Array in Node, Deno, Bun and the browser. Buffer is accepted, never required.
• Source-agnostic, copy-free. A Float32Array is its bytes. A DataView is its bytes. An ArrayBuffer is all of its bytes. An Array<number> is byte values. Pieces come back as subarray views of the caller's memory.
• Lazy and bounded by default. Iterators stop when you stop pulling. max lets you ask "more than N distinct bytes?" and leave. find scans once for any number of needles.
• Honest about kind. Predicates return 0 / 1 / -1: the sign tells you whether the two sources are the same kind of object, so "same bytes" and "same bytes and same type" are one call apart.
• Pure ESM, dependency-free, sync. One file, ~1350 lines, Node ≥ 20. Hand-authored TypeScript declarations.

What it is NOT

• Not a faster Buffer. Buffer.equals, Buffer.compare and zlib.crc32 are C++ and they win on the operations they cover, by 10–25×. buf-im wins where the native idiom is forced to do work you didn't ask for — allocate every line to read the first, scan once per needle, copy to trim — and it wins everywhere Buffer does not exist. See Performance.
• Not a text library. Bytes are not characters. UTF-8 validation, grapheme counting and byte-safe truncation of text live in @slnknrr/str-im, which reads utf-8 buffers directly. A string is accepted here only as a needle, meaning its utf-8 bytes.
• Not an element-wise typed-array library. A Uint16Array([1]) is the two bytes 01 00, and buf-im says so. Comparing typed arrays by element value belongs to @slnknrr/arr-im.
• Not hex / base64. Those are a standard proposal (Uint8Array.prototype.toHex, fromBase64) and Node has Buffer for them; duplicating them here would be a lie about what is missing.
• Not cryptography. crc32, adler32 and fnv1a are integrity checks and hash-table hashes. teq is the one security-minded method, and its contract is stated precisely below.

Install

npm install @slnknrr/buf-im

import b from '@slnknrr/buf-im';                     // ready-to-use singleton — no `new`
import { bufim, _bufim } from '@slnknrr/buf-im';     // the class (for use() / instanceof) and the factory

Requirements: Node ≥ 20, ESM only ("type": "module" or import).

Core conventions

These are load-bearing. Learn them once; they apply everywhere.

A source is a run of bytes. Whatever object you pass, the engine reads bytes:

| Source | Read as | |---|---| | Uint8Array, Buffer | as it is | | any other ArrayBufferView — Int8Array, Uint16Array, Float32Array, BigInt64Array, DataView, … | the bytes it sits on: [byteOffset, byteOffset + byteLength) | | ArrayBuffer, SharedArrayBuffer | all of its bytes | | Array<number> | byte values, coerced exactly as new Uint8Array(array) coerces them (& 0xff, NaN → 0) |

No element types, no endianness of the source's own elements. A string is not a source (TypeError); as a needle, a separator or a fill pattern it means its utf-8 bytes.

Predicates return 0 | 1 | -1; the sign is the kind. 0 = no. 1 = yes, and both sources are of the same kind (the same constructor). -1 = yes, but the kinds differ. -1 is truthy, so if (b.eq(a, c)) reads naturally — and "same bytes" stays distinguishable from "same bytes and same type" without a second call.

b.eq(u8, new Uint8Array([1, 2, 3]));   // 1   same bytes, both Uint8Array
b.eq(u8, Buffer.from([1, 2, 3]));      // -1  same bytes; a Buffer is another kind
b.eq(u8, new DataView(u8.buffer));     // -1  same bytes through a DataView
b.eq(u8, [1, 2, 3]);                   // -1  same bytes as an array
b.eq(u8, [1, 2, 4]);                   // 0

cmp is the one exception: it is a comparator (-1 / 0 / 1 by content, like memcmp) and ignores kind.

Validation is synchronous. A generator's body doesn't run until the first next(), so a naive lazy API would throw later, in the consumer's stack. buf-im primes every iterator: all argument checks happen at the call site.

b.find(buf, []);          // throws TypeError NOW, not on first iteration: an empty needle

Errors are typed. TypeError = wrong type (an unsupported source, a string where bytes were needed, a read-only source in a writer). RangeError = an offset, width, count or value out of range, or no room to write.

Writers fail before the first byte. The in-place family and the setters check everything up front, so a source is never left half-written. They return the target (the variable-width setters return the end offset).

Saturating counts return Infinity. b.ulen(buf, 16) returns Infinity once 16 distinct bytes are seen — an honest "at least this many", never confused with a real count of 16.

Bits are numbered MSB-first, integers are big-endian, as in every RFC diagram. { lsb: true } and { le: true } flip that per call. LEB128 is little-endian by definition and takes no option.

Sources & adapters

Every reading method resolves an adapter once per call, then stays in a tight numeric loop. The four kinds above work out of the box; a non-Uint8Array view is normalized to a Uint8Array overlay at the boundary — one small wrapper object, never a copy of the bytes.

Register your own type once, process-wide, and the entire method family works with it:

bufim.use(Ring, {
  size: (r) => r.length,            // number of bytes
  at:   (r, i) => r.byteAt(i),      // the byte at i, 0..255
  put:  (r, i, v) => r.write(i, v), // optional: without it the type is read-only
  view: (r) => r.contiguous(),      // optional: a Uint8Array over the same memory, or null
});

With a view, split, chunk, trim and bytes hand out zero-copy pieces of your memory; without one they hand out copies, and overlap treats the type as sharing memory only with itself.

Performance

buf-im is fast where laziness, single-pass multi-pattern search and not copying matter. It does not compete with V8's C++ on the operations Buffer already covers, and says so out loud.

Representative run (npm run bench, Node 24, one machine — your numbers will differ; the shape won't):

| Case | buf-im | the native way | speedup | |---|--:|---|--:| | first line of a 110 KB log | 980k/s | decode + split('\n')[0] | ~220× | | trim 64 KB | 64M/s | decode + trim() | ~700× | | entropy of 64 KB | 14k/s | Map counts + reduce | ~24× | | distinct byte values of 64 KB | 10k/s | new Set(bytes).size | ~21× | | read a 48-bit big-endian int | 51M/s | two DataView reads + math | ~4.5× | | find any of 400 needles, one pass | 660/s | 400 × Buffer.indexOf | ~1.5× | | concatenate 64 × 1 KB views | 16k/s | Buffer.concat | ~0.8× (par) | | crc32 of 64 KB | 6k/s | zlib.crc32 (C) | 0.12× | | eq of two 64 KB buffers | 13k/s | Buffer.equals (C) | 0.04× |

Why the wins are structural: split yields the first line as a view instead of decoding and allocating all 2000; trim returns a subarray; ent and ulen use a 256-slot table instead of hashing; uint reads six bytes in one loop; Aho-Corasick scans once at a cost flat in the needle count, while indexOf — fast as it is — pays one full scan per needle and pulls ahead only while the needle list is short. The last two rows are memcmp and a CRC in C; nothing in JavaScript beats them, and buf-im exists for the runtimes and the operations where they are not there.

API

66 methods across 10 groups. buf and src accept any source; dst must be writable (any built-in source is). needle is a byte value, a string (its utf-8 bytes) or a source; needles is one of those or an iterable of them — an Array of numbers is one needle, an Array of anything else is a collection. Iterator methods return lazy IterableIterators.

1 · Predicates — 0 / 1 / -1, the sign is the kind

| Method | Asks | |---|---| | eq(a, b) | the same bytes? | | ne(a, b) | different bytes? — ne(a, b) === 0 exactly when eq(a, b) !== 0 | | ge(a, b) | does a start with b? (prefix match + a at least as long) | | le(a, b) | is a a prefix of b? | | cmp(a, b) | -1 / 0 / 1 by unsigned bytes, then by length — memcmp, a comparator for sort; kind ignored | | teq(a, b) | the same bytes, in constant time |

teq reads every byte of both sources and folds the differences into one accumulator, so the time does not depend on where they differ. Lengths and kinds are not secrets: a length mismatch returns 0 at once, as crypto.timingSafeEqual would throw.

2 · Search & compare — offsets in bytes

| Method | Returns | |---|---| | find(buf, needles, {all}?) | lazy match offsets, left to right; many needles in one pass | | lfind(buf, needles, {all}?) | right to left | | com(a, b) / comb(a, b) | length of the common prefix / suffix (b = one source or many) | | ham(a, b) | Hamming distance in bits between two sources of equal length |

Matches are non-overlapping and leftmost-longest: with needles ab and abc, abcd yields one match, the longer; with abc and c, abcabc yields [0, 3], never [2, 5]. { all: true } reports every match, nested and overlapping ones included, in order of completion. The same rule drives split: split(buf, ['::', ':']) cuts on :: first, whatever the order you list them in.

3 · Counters — no allocation, one table at most

| Method | Counts | |---|---| | ulen(buf, max=∞) | distinct byte values (a 256-bit bitmap); Infinity at max | | popcnt(buf) | set bits | | clz(buf) / ctz(buf) | leading / trailing zero bits | | ent(buf, base=2) | Shannon entropy per byte: 0 constant, 8 uniform; compressed data sits near 8, text near 4–5 | | freq(buf) | Uint32Array(256) — the histogram itself; the one deliberate allocation | | sum(buf) / xsum(buf) | arithmetic sum / xor of all bytes (the one-byte checksum of NMEA and serial protocols) |

4 · Runs — 00 00 00 ff ff is two runs

| Method | Returns | |---|---| | consc / rconsc (buf, max=∞) | length of the first / last run; Infinity at max | | cons / rcons (buf, max=∞) | the byte of each run, from the start / the end | | consg / rconsg (buf, max=∞) | the length of each run | | ubyte(buf, max=∞) | distinct byte values, first-seen order |

5 · Bits — MSB-first; {lsb: true} for LSB-first

| Method | Does | |---|---| | bit(buf, i, opts?) | bit i as 0 / 1 | | bits(buf, off, n, opts?) | n bits (1..53) from bit offset off, as a number — across byte boundaries, unaligned | | setbit(buf, i, v, opts?) | set bit i in place; returns buf | | setbits(buf, off, n, value, opts?) | write an n-bit field in place; RangeError before any write if value does not fit |

b.bits(hdr, 4, 12);                 // a 12-bit field that starts in the middle of byte 0
b.bits(deflate, 0, 3, { lsb: true }); // formats that fill bytes from the low end

6 · Integers of any width — big-endian; {le: true} for little-endian

| Method | Returns | |---|---| | uint / int (buf, off, n, opts?) | an n-byte unsigned / two's-complement integer, n ≤ 6, as a number (48 bits are always exact) | | buint / bint (buf, off, n, opts?) | the same for any n, as a BigInt — 16-byte UUIDs, 32-byte hashes | | putuint / putint (buf, off, n, value, opts?) | write in place: a number for n ≤ 6, a BigInt for any n; a value that does not fit throws before writing; returns buf | | uvar(buf, off) / svar(buf, off) | an unsigned / zigzag-signed LEB128 varint (protobuf, wasm, git); uvar throws past 2⁵³ − 1 | | buvar(buf, off) | an unsigned varint of any length, as a BigInt | | varlen(buf, off) | the encoded length of the varint at off | | varsize(value, {signed}?) | bytes needed to encode value | | putuvar / putsvar (buf, off, value) | write a varint; returns the offset just past it — the next field starts there |

let off = 0;
off = b.putuvar(out, off, fieldTag);
off = b.putuvar(out, off, payload.length);
b.uint(ulid, 0, 6);                  // the 48-bit millisecond of a ULID, without BigInt
b.buint(uuid, 0, 16).toString(16);   // a UUID as one 128-bit number

7 · Checksums — integrity, not security

| Method | Returns | |---|---| | crc32(buf, {init}?) | CRC-32 (IEEE 802.3): zip, png, gzip, Ethernet — the number zlib.crc32 gives | | adler32(buf, {init}?) | Adler-32 (zlib) | | fnv1a(buf, {bits: 32 \| 64}?) | FNV-1a: 32 bits as a number, 64 as a BigInt — hash tables, bloom filters |

init chains a stream piece by piece: b.crc32(rest, { init: b.crc32(head) }) equals the CRC of head followed by rest. The CRC table is built on first use.

8 · Views & splitting — pieces without copying

| Method | Returns | |---|---| | bytes(src) | a Uint8Array over the same memory of anything that has it (a Float32Array's bytes, a DataView's window, an ArrayBuffer); an array comes back as a copy | | split(buf, sep?, {limit}?) | lazy zero-copy pieces; sep absent ⇒ line breaks (\r\n | \n | \r as one) | | splitn(buf, sep?, {limit}?) | boundaries instead of pieces: a flat stream start0, end0, start1, end1, … — parsing with no allocation | | chunk(buf, size) | lazy zero-copy pieces of size bytes, the last one shorter | | chunkn(buf, size) | the boundaries 0, size, 2·size, …, length; piece k is [b[k], b[k+1]) | | trim / ltrim / rtrim (buf, bytes?) | strip a byte set from the edges, as a view; ASCII whitespace by default; the source itself when nothing was trimmed | | cat(bufs) | concatenation into one new Uint8Array of exactly the right size — Buffer.concat without Buffer | | overlap(a, b) | do two sources share memory? 0 no · 1 the very same range · -1 an overlap that is not the same range |

Pieces are subarray views for anything with memory and slice copies for an Array. sep and bytes take a byte, a string (its utf-8 bytes), a source, or — for sep — an iterable of them.

9 · In place — the only writers besides the setters

| Method | Does | |---|---| | xor / and / or (dst, src) | dst[i] op= src[i mod src.length] — the pattern repeats: a mask, a one-time pad, a WebSocket frame key | | not(dst) | invert every bit | | fill(dst, pattern) | overwrite with a byte, a string (utf-8) or a source, repeated — Buffer.fill(string) for any source | | swap(dst, width) | reverse every width-byte word — the endianness swap of a whole array, Buffer.swap16/32/64 for any width |

Each writes the caller's own memory, allocates nothing, and returns dst.

10 · Detection & second tier

| Method | Returns | |---|---| | bom(buf) | byte length of a leading BOM: 3 utf-8 · 2 utf-16 · 4 utf-32 · 0 none | | magic(buf) | the file format named by the signature — png jpeg gif bmp tiff ico webp avif heic jxl psd pdf zip gzip bzip2 xz zstd lz4 7z rar ar tar iso elf exe macho wasm sqlite mp3 flac ogg wav avi mp4 mov matroska flv woff woff2 ttf otf rtf xml pcap pcapng — or undefined | | dump(buf, {width, offset}?) | hexdump -C lines, lazily: offset, bytes grouped by eight, printable ASCII in bars | | hash(buf, n) | rolling Rabin–Karp hash of every n-byte window, the next from the previous in O(1) — content-defined chunking, dedup |

[...b.dump(header)].join('\n');
// 00000000  89 50 4e 47 0d 0a 1a 0a  00 00 00 0d 49 48 44 52  |.PNG........IHDR|
// 00000010  00 00 01 00 00 00 01 00  08 06 00 00 00           |.............|

magic names what the header claims; it is a sniff of the first bytes, not a parser.

Statics & factory

| Symbol | Purpose | |---|---| | bufim.use(ctor, adapter) | register a source type | | _bufim(overrides?) | build a configured instance (see below) | | bufim | the class, for use() and instanceof |

Extending & configuring

The default export is a singleton — you never write new. To specialize behavior, pass overrides to _bufim. An override may call super to wrap the original:

import { _bufim } from '@slnknrr/buf-im';

const traced = _bufim({
  find(buf, needles, opts) {
    console.count('find');
    return super.find(buf, needles, opts);
  },
});

Every internal call dispatches through this, so replacing one method replaces it everywhere it is used.

⚠️ Keep configurations few and long-lived

Internal call sites are monomorphic and V8 inlines them to zero cost — as long as few distinct buf-im classes exist in the process. Create your configuration once, at module load, and reuse it. _bufim caches by the overrides object (a WeakMap), so the same object always yields the same instance. Spawn more than ~4 distinct configurations and the loops that make this library cheap go megamorphic.

Design notes

• Normalize at the boundary, not in the loop. A Float32Array or an ArrayBuffer becomes a Uint8Array overlay once, before the adapter is picked, so the hot loops see two adapters at most: Uint8Array and Array. The kind used by the predicates' sign is taken from the argument as given, before normalization.
• The hot loops are plain methods, not generators. V8 optimizes a generator's body far less aggressively than an ordinary function; a per-byte loop inside one ran six times slower than the same loop outside. So find, split and friends are generators that only hand out results, each driven by a scanner that runs the tight loop and returns the next hit. One idle yield per call for synchronous errors, nothing per byte.
• Leftmost-longest, properly. Aho-Corasick reports a match when it completes, which is not when it starts. A candidate is held until no other match can start at or before its start (start + maxLen − 1 bytes later), then emitted, and the scan restarts at its end — re-reading at most maxLen − 1 bytes per match, allocating nothing. With one needle the candidate settles the moment it completes.
• A dense DFA when it fits. Automata up to 4096 nodes get a Uint16Array transition table with the fail links resolved at build time — one array read per byte. Larger sets keep sorted sparse transitions and a binary search. Both are cached per needle set, LRU, 16 deep.
• Tables, once. The per-byte popcount table is 256 bytes at load; the CRC-32 table (1 KB) exists only after the first crc32().

Scripts

| Command | Does | |---|---| | npm test | behavioral suite (node --test, zero dependencies) — against Buffer, DataView, zlib.crc32, timingSafeEqual and published vectors, plus a brute-force differential test of the search | | npm run types | type-check the shipped declarations (tsc --noEmit) | | npm run bench | the benchmarks above |

Links

• Source: https://codeberg.org/slnknrr/bufim_lib.js-for-pm_npm/src/branch/main
• npm: https://www.npmjs.com/package/@slnknrr/buf-im
• Text over the same buffers: @slnknrr/str-im · structure over arrays: @slnknrr/arr-im

Author

Yury Slinkin (Юрий Слинкин)

• Email: [email protected]
• Codeberg: https://codeberg.org/slnknrr

License

MIT. See LICENSE.md.