@qezor/chunkysys

@qezor/chunkysys is a low-memory chunk processing system for files sitting on top of fs-like object storage adapters such as @qezor/objsys.

It is meant to be the foundation for:

• bounded file transforms
• line-based rewrites
• text streaming
• delimiter and paired-frame iteration
• guard-controlled buffer lifetimes
• live rewriters later

It is not a scoped parser like jsonsys. It stays generic and works on bytes, text, and line streams.

What It Prioritizes

• bounded buffer budgets
• chunk leases that release memory as soon as they are no longer needed
• async iterator APIs instead of callback pyramids
• no recursion-heavy internals
• direct streaming into object storage writes
• precise byte windows via startByte and endByte
• range-aware source reads when the underlying fs adapter exposes supportsRanges

Install

npm install @qezor/chunkysys

Main Helpers

• createChunkySystem(options)
• createChunkySystemFromEnv(options)
• createChunkGuard(options)
• chunks(path, options)
• text(path, options)
• lines(path, options)
• delimit(path, options)
• frames(path, options)
• copy(sourcePath, destinationPath, options)
• rewriteBytes(sourcePath, destinationPath, transformer, options)
• rewriteText(sourcePath, destinationPath, transformer, options)
• rewriteLines(sourcePath, destinationPath, transformer, options)

Core Idea

chunkysys does not try to store whole files in memory.

Instead it:

• reads from an fs-like stream
• exposes a bounded chunk lease
• releases that chunk automatically on the next step unless you explicitly retain it
• streams transformed output straight back into the destination writer

That makes it usable in tight environments where holding the whole file is not acceptable.

Usage

const { createChunkySystem } = require("@qezor/chunkysys")

const chunky = createChunkySystem({
  maxBufferBytes: 64 * 1024,
  chunkBytes: 16 * 1024,
})

for await (const chunk of chunky.chunks("catalogs/v1.txt")) {
  console.log(chunk.index, chunk.byteStart, chunk.byteEnd, chunk.text())
}

Text And Line Rewrites

rewriteText() and rewriteLines() accept either:

• a function
• a small text plan object or array of plans

The plan path uses @qezor/structkit string helpers under the hood.

Example:

await chunky.rewriteLines(
  "drafts/input.txt",
  "drafts/output.txt",
  [
    { op: "insert", after: "bulk", value: "-" },
  ]
)

Precise Byte Windows

You can limit reading to a byte range without loading the whole object:

for await (const chunk of chunky.chunks("big.bin", {
  startByte: 1024,
  endByte: 4096,
  chunkBytes: 512,
})) {
  // process only the requested window
}

Delimiters And Frames

delimit() is for precise streaming splits without loading the full object:

for await (const item of chunky.delimit("queues.ndjson", {
  delimiter: "\n",
  maxSegmentBytes: 128 * 1024,
})) {
  console.log(item.text)
}

frames() is for paired delimiters such as (), <>, {{ }}, or block markers:

for await (const item of chunky.frames("template.txt", {
  open: "{{",
  close: "}}",
  emitOutside: true,
  allowNested: true,
})) {
  console.log(item.kind, item.text)
}

Guard System

createChunkGuard() gives you a resource governor above plain byte budgeting:

• byte limits
• step limits
• rules
• lifetimes
• scheduled purging

Example:

const { createChunkGuard, createChunkySystem } = require("@qezor/chunkysys")

const guard = createChunkGuard({
  maxBytes: 64 * 1024,
  maxSteps: 10_000,
  autoPurgeEverySteps: 100,
  defaultLifetimeMs: 5_000,
  purgeIntervalMs: 1_000,
})

const chunky = createChunkySystem({
  guard,
})

Notes

• retain() is for rare cases. If you retain too much and your budget is exhausted, the system will fail fast by default.
• retained chunks can now be given explicit lifetimes, and the guard can purge expired resources on schedule or step intervals
• when the source adapter supports ranged reads, byte-window iteration avoids pulling unrelated bytes through the stream path
• rewriteBytes() is the right base for binary or parser-specific transforms later.
• rewriteLines() is a good fit for markdown, config, logs, NDJSON, and annotation passes.
• this package is designed to be the low-memory base for later DOM parsers and LiveRewriter systems

Versioning

@qezor/chunkysys follows practical semver:

• patch: safe fixes, doc cleanup, and low-risk performance or correctness improvements
• minor: additive stream, guard, or rewrite capabilities that do not break existing callers
• major: contract changes to chunk leases, iterator shapes, guard semantics, or rewrite behavior

Release notes live in CHANGELOG.md.