npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2026 – Pkg Stats / Ryan Hefner

@clinth/blocktransfer

v0.2.2

Published

Verified, resumable filesystem and direct SMB block transfers

Readme

blocktransfer

A portable byte-transfer engine with Node.js filesystem, direct SMB, directory and CLI adapters. Provider integrations use transferBytes with caller-owned storage, checkpoints and locking. The existing path/CLI APIs retain adaptive copying into private staging and full read-back verification by default.

Requires Node.js 22.12 or later. The package is ESM, with TypeScript types and JSDoc included. Peer networking belongs in separate packages; see library boundaries and cross-device integrations. See usage and correctness limits for recovery details and platform constraints, and the implementation checklist for project status.

Opt-in SMB/mounted-volume tests and native macOS performance comparisons accept any explicitly selected volume. Run npm run test:volume -- --volume /Volumes/Temp or npm run bench:volume -- --volume /Volumes/Temp; these create and clean up only uniquely owned test directories. Ordinary tests stay inside the repository.

To compare a mount with direct smb3-client access to the same share, use the SMB access benchmark, which reports storage I/O separately from full blocktransfer copies and records cache/security/publication differences.

Library usage

For provider/daemon integrations, use transferBytes and the store contracts from @clinth/blocktransfer/core (also exported at the root). This entry has no Node imports, sidecars, directory planner or subprocesses. Its default verification is inline; select readback for an independent source/staging pass. The @clinth/blocktransfer/local entry supplies sidecar-free path/SMB convenience with caller-owned checkpoints. Existing root path APIs retain their recovery defaults for compatibility. See the request-by-request status.

Further entries: @clinth/blocktransfer/http (createHttpSource, a range-reading source with Node connection control; the same function is in /core with an injectable fetch) and @clinth/blocktransfer/testing (the store contract suites and fault-injection helpers). The root and /local also export createSourceStore and createSinkStore. See provider-stores.

Import from @clinth/blocktransfer when installed as a dependency. When running examples inside this checkout, build with npm run build and use ./dist/index.js instead. The examples assume the input paths already exist and the output files do not.

Paths, file URLs and direct SMB URLs

transferFile, copy and selectEntries accept native paths, URL strings and URL objects. file:// URLs use the local filesystem; smb:// URLs connect directly through our ../smb3-client fork, with no mounted share required. The first SMB path segment names the share. Source and destination can use different backends, hosts or shares.

await transferFile(new URL('file:///Users/me/video.mp4'),
  'smb://user:password@fileserver/temp/video.mp4');

// Omit the password to use a matching system credential.
await copy('smb://user@fileserver/temp/photos', './photos', { recursive: true });

// Applications can supply their own secret-store integration.
await transferFile('./video.mp4', 'smb://fileserver/temp/video.mp4', {
  smb: { credentials: async ({ host, share, username }) =>
    myCredentialStore.lookup({ host, share, username }) },
});

Percent-encode usernames, passwords and individual path segments containing reserved characters; setting URL.username and URL.password also encodes them. DOMAIN;user or DOMAIN\user account forms are supported. An explicit password (including an empty one) takes precedence over credential lookup. Quote URL arguments in the shell. Inline credentials can appear in shell history/process arguments; omit them when using a secret store.

System credential lookup (opt in with smb.systemCredentials: true; the CLI enables it unless --no-keychain) uses macOS SMB internet-password entries, Linux Secret Service via secret-tool (include the username in the URL), or Windows generic credentials named blocktransfer:smb://host/share (include a nondefault port when used). Windows domain credentials are not treated as exportable passwords. The library does not read the keychain by default. Stores may require unlocking. Blocktransfer does not save passwords to recovery state, progress, results or errors, and it does not create credentials. See credential-store details.

Direct SMB retains block resumption, independent read-back, conflict checks and atomic publication. Server-held exclusive handles coordinate competing SMB URL writers. auto uses block copying; native/ditto/rsync strategies require filesystem paths. SMB preserves modification times and the read-only attribute, rather than all POSIX permissions or ACLs. File data is flushed; portable SMB directory fsync is unavailable. SMB signing is required by default; encryption is off by default; use smb.encryption: 'if-offered'/'required' (--smb-encryption) to enable SMB3 transport encryption. Cancellation lets pending SMB requests settle, interrupting stalled connections after 500 ms. Uncertain remote outcomes retain recovery evidence.

Transfer a file

import { transferFile } from '@clinth/blocktransfer';

const result = await transferFile('./input/video.mp4', './output/video.mp4');
console.log(`Completed ${result.size} bytes in ${result.elapsedMs} ms`);
console.log(`Copied ${result.copiedBytes}; reused ${result.reusedBytes}`);

The destination is an exact filename; parent directories are created as needed. Existing files are refused by default. To intentionally replace one, pass { overwrite: true }. Basic permissions and timestamps are preserved; ownership, ACLs and extended attributes are not transferred.

Copy directory contents or select files

import { copy } from '@clinth/blocktransfer';

// Copy all contents, including nested files and empty directories.
const files = await copy('./input', './output', { recursive: true });
console.log(`Completed ${files.length} files`);

// Copy matching files into another destination, keeping their relative paths.
const selected = await copy('./input', './text-output', {
  glob: ['**/*.txt', '**/*.md'],
  maxDepth: 3,
});
console.log(`Completed ${selected.length} matching files`);

Directory sources copy their contents into the destination. Without selection options, only immediate files are copied. Globs enable traversal as needed and use / separators on every platform; matching dotfiles are included. maxDepth: 0 selects immediate files, 1 includes files in immediate subdirectories, and so on. Empty directories are preserved when traversing without globs.

copy also accepts a file source and returns a one-element result array. Directory results contain one entry per file, including unchanged files skipped on resume. Unrelated destination files are preserved. Directory copies publish one file at a time and require non-overlapping source/destination trees. Selected symlinks and special files are unsupported; use a stable source tree rather than expecting a snapshot of a changing directory.

Preview a selection without copying anything:

import { selectEntries } from '@clinth/blocktransfer';

for await (const entry of selectEntries('./input', { glob: '**/*.txt' })) {
  console.log(entry.kind, entry.relativePath);
}

Observe progress, cancel and resume

import { transferFile, TransferError } from '@clinth/blocktransfer';

const controller = new AbortController();
// A UI cancel action could call controller.abort(). This example uses a timeout.
const timeout = setTimeout(() => controller.abort(), 5_000);
let paused = false;

try {
  await transferFile('./input/archive.bin', './output/archive.bin', {
    signal: controller.signal,
    onProgress(progress) {
      console.log(progress.method, progress.phase, progress.processedBytes, progress.totalBytes);
      if (progress.etaSeconds !== null) {
        console.log(`Copy pass: about ${Math.ceil(progress.etaSeconds)} seconds left`);
      }
    },
  });
} catch (error) {
  if (error instanceof TransferError && error.code === 'ABORTED') {
    paused = true;
    console.log('Paused; staged data is retained.');
  } else {
    throw error; // Includes native filesystem errors such as ENOSPC.
  }
} finally {
  clearTimeout(timeout);
}

if (paused) {
  // Resume now or in a later process, using the same paths and a fresh signal.
  const result = await transferFile('./input/archive.bin', './output/archive.bin');
  console.log(`Reused ${result.reusedBytes} bytes`);
}

Callbacks are synchronous and should return quickly. Progress is per file, emitted at phase changes, block boundaries and approximately every second during pending asynchronous I/O. processedBytes is the copy/compare pass; verifiedBytes tracks the separate read-back pass. Copy/reuse counters are logical bytes, not physical disk I/O. ETA excludes verification and is null outside copying.

For directory totals, supply onDirectoryProgress(progress) to copy. It reports enumeration, completed/active files, processed/verified bytes, rate and copy ETA. The CLI displays this aggregate view automatically. Native copiers emit a heartbeat with bytesKnown: false while byte progress is unavailable.

Every file progress event includes method: pending, block, native, ditto, rsync, rsync-batch, smb or skipped. Directory events include the active method (or mixed) and a methods list. Rsync also reports transferredBytes; these bytes can still be buffered and are separate from completed copying, verification and publication.

Call controller.abort() to stop. Default cancellation retains resumable staging; use cleanupOnCancel: true (CLI --cleanup-on-cancel) to discard owned unpublished data and recovery records. Disposable batch work trees are always removed after the copier closes. Already published files and uncertain publication evidence are retained. Empty coordination directories may remain. See cancellation limits.

Recovery and verification are independently configurable with { resumable: false } and { verify: false }, or CLI --non-resumable and --non-verified. Both remain enabled by default. engine: 'auto' (the default; strategy is the older name) learns from completed files in the current process. You can name an engine instead: block, fs, cp, cpy (optional npm module), ditto (macOS), rsync, or smb2 (the smb2 command line tool, direct SMB). listEngines() / --list-engines shows what works on your machine; see Engines.

Library users get onEvent lifecycle events (started, progress, then one of finished, paused or error), hidden recovery markers at the destination and source, and findPendingOperations, probeOperation, resumeOperation and cancelOperation to pick up unfinished work at startup; see Lifecycle events and pending operations. See mode semantics and adaptive limits. The backend investigation compares rsync with equivalent guarantees and assesses direct SMB publication.

On macOS/Linux, automatic directory copying can batch eligible small files through rsync, then compare completed batch timings with the built-in engine. Both paths retain the requested recovery and verification guarantees. Rsync is optional: missing/incompatible executables fall back to built-in copying. Use allowRsync: false / --no-rsync to prohibit it, or rsyncPath / --rsync-path to select an executable. Windows automatic copying uses the portable engine. The cross-device investigation records locality and transport experiments that can inform a separate networking package.

Resume directory operations with the same paths and selection options. Source changes pause by default; { follow: true, followRetryMs: 250 } retries until the source settles or the signal is cancelled, then returns after a successful copy. Destination changes after starting are a separate conflict: overwrite alone does not discard them. Only pass overwriteDestinationChanges: true after reviewing those edits and deciding to replace them.

API reference

All exports below are available from blocktransfer; types are TypeScript-only.

| Export | Purpose | | --- | --- | | transferFile(source, destination, options?) | Transfer one file; returns Promise<TransferResult>. | | copy(source, destination, options?) | Transfer a file or directory contents; returns Promise<TransferResult[]>. | | selectEntries(root, options?) | Preview selected entries as an async iterator of SelectedEntry. | | stateDirectory(destination) | Compute the absolute local path or credential-free SMB URL for recovery, without I/O. | | TransferError | Library error with a machine-readable code; native I/O errors also propagate. | | TransferOptions, SelectionOptions, CopyOptions | Transfer controls, selection rules, and their combination. | | TransferLocation, SmbOptions, SmbCredentials, SmbCredentialRequest, SmbCredentialProvider, systemSmbCredentials(request) | Resource inputs, direct SMB policy and system/custom credential resolution. | | Progress, DirectoryProgress, TransferPhase, TransferMethod, TransferResult, SelectedEntry, TransferErrorCode, PublicationAdapter | Typed progress, copier identification, results, selections, failures and an explicit atomic publication capability. |

blockSize defaults to 1 MiB (maximum 64 MiB); checkpointBlocks defaults to 16. Resume reuses the saved block size; explicitly supplying a different size is an error. Recovery directories live beside destinations and may remain empty after success. Do not edit recovery data while a transfer is active.

Verification requires an unchanged observed source; it cannot provide a coherent snapshot of an arbitrarily changing file. Publication and crash durability depend on filesystem/OS support. Read the platform limits before relying on the library for concurrent writers or power-loss guarantees.

Library boundaries and cross-device integrations

Blocktransfer owns transfer correctness and efficient work against storage: selection, hashing, block reuse, staging, recovery, conflict detection, verification, publication and progress. SMB is a storage backend, using the local fork for its protocol, authentication and connection security. Peering, client/server services, discovery and peer transfer protocols belong in another package. That package could use WebSockets, iroh, HTTP or another data backend without making any of them a requirement for local copying.

The intended design lets that package invoke blocktransfer beside each resource. Source hashing stays beside the source; destination comparison, staging and read-back verification stay beside the destination. Only manifests, missing data and status need to cross the network. Merely putting a remote filesystem API under the current copy loop would retain expensive remote reads and round trips.

Available today

The supported entry points are the API reference above. Native paths and file:// URLs access the calling process's filesystem; smb:// URLs access SMB storage directly. The built-in SMB adapter still reads verification bytes across SMB in the calling process.

transferBytes, SourceStore, SinkStore, Checkpoint and StateStore are public provider integration APIs. A separate package can implement source/sink operations beside its storage, using copyRange, source.digest and sink.digest to keep payload processing in its provider process. Checkpoints and locks are caller-owned; the core imposes no path or transport. See the provider contract. This supplies a single-item engine, not an exported peer job protocol or manifest scheduler.

TransferOptions.publicationAdapter is an existing, narrow integration hook. Its renameExclusive(staging, destination) operation must atomically publish the same staged object, preserve its identity/metadata, refuse an existing destination with EEXIST, and wait until the mounted filesystem observes the rename. An adapter must establish that both paths belong to the same storage namespace. Errors propagate without a fallback because publication may already have happened. This hook applies to new destinations; replacements use filesystem publication. It is useful for storage-specific optimizations, not a general source, sink or transport interface. Credentials and callback state are not saved in recovery data.

Intended extension points

The broader paged-session/planner boundaries below remain a design, beyond the public single-item store interfaces above. Internal legacy manifests, filesystem helpers and directory journals are not supported integration APIs.

| Component | Responsibility in blocktransfer | | --- | --- | | Source session | Enumerate selected entries, observe source identity/generation, compute paged hash manifests, read requested ranges and revalidate the source before completion. An observed generation does not imply a filesystem snapshot. | | Block planner | Compare source digests with locally validated destination/staging digests and return bounded pages of missing ranges. Keep this computation independent of data delivery. Start with the existing fixed-offset blocks and SHA-256; shifted-block matching is separate work. | | Destination session | Own locks, private staging, conflict policy and recovery state; accept validated blocks, checkpoint, independently read back storage, and publish with a recoverable completion receipt. | | Storage capability adapter | Supply local positioned I/O, identity observations, staging, flush, metadata and publication operations. Explicitly describe capabilities so a filesystem or future embedded-device backend cannot silently weaken the requested guarantees. | | Progress and cancellation | Expose bounded asynchronous work and phase events for hashing, comparison, writing, verification and publication. Preserve cooperative cancellation and the default resumable, verified behavior. |

The separate integration package owns endpoint lifecycle, path/export authorization, authenticated manifest exchange, protocol versioning, transport framing, retries, connection changes and data scheduling. It translates core job operations and events into its own protocol. Blocktransfer should not require a socket, peer identity or connection ID to open or resume a storage job. Transport frame sizes and stream counts must remain independent of content block sizes.

A cross-device integration would use these boundaries as follows:

  1. Open a source session and send its manifest pages, binding file identities, sizes, block layout, digests and the observed source generation to the job.
  2. Open a destination session with an explicit conflict policy. Hash existing or staged data locally and return the missing ranges; saved checkpoints alone are not proof that bytes still match.
  3. Deliver only the needed blocks through the chosen data backend. Validate each block's job/generation, range, length and digest. Bound outstanding bytes and let destination storage pressure control backpressure.
  4. Revalidate the source locally and independently read back and hash destination staging locally against the bound manifest. Hashing received buffers alone does not replace filesystem read-back. Changed sources or destinations pause the job unless an explicit policy permits reconciliation or replacement.
  5. Publish through the destination session and obtain its completion receipt. Distinguish received, written, durably checkpointed, verified and published states. After a lost reply, query/reconcile the job instead of blindly repeating publication. A successful socket send is not a durable checkpoint.

This can reduce network traffic without removing verification: a fresh file need not be sent back to its source for read-back hashing, and a resumed transfer need only send missing blocks. Both endpoints still perform local reads. Actual speed depends on disk, hashing CPU, transport and workload; the locality experiments are loopback research, not a cross-device throughput guarantee. Random sampling may detect differences early, but cannot prove equality or replace full verification by default.

Extraction should preserve the existing local API and its recovery/conflict tests, then expose independently testable source, planner and destination operations. A separate package can build networking around those operations; this package's CLI remains a filesystem/SMB copier. The same storage jobs should remain usable without a network stack, with explicit resumable: false and verify: false policies available when requested.

CLI and development

npm ci
npm run check
node dist/cli.js --help
node dist/cli.js ./input/video.mp4 ./output/video.mp4
node dist/cli.js --glob '**/*.txt' ./input ./output

Ctrl+C pauses a transfer; run the same command to resume. Quote globs so the shell does not expand them. See the usage guide for more CLI options, conflict handling, tests and benchmarks.

Compare copying performance

npm run bench:compare -- --mib 256 --files 1000 --repeats 3
# Or select a mounted destination:
npm run bench:compare -- --volume /Volumes/Temp --mib 64 --files 200

Writes a Markdown report and raw JSON under benchmarks/results/, comparing blocktransfer's four resume/verification modes against cp, ditto and rsync on small files, a large file and partial-file resume. Native verified variants include the time to launch a command and freshly hash both source and destination. An additional correctness audit is reported separately. See benchmark options and timing semantics.

Original project brief

Historical context follows. The package boundary above supersedes suggestions below to include peering or client/server services in blocktransfer itself.

This is a library (and separate CLI) to do block-based transfer of data within and between filesystems and devices.

The emphasis is on speed, reliability and accuracy.

At a basic level it should support transfering a file from point A to B. If anything goes wrong, it should support resumability, guaranteeing that the transfer is byte-perfect and with minimal amount of repeated data transfer. Similar to what rsync can offer. For example, starting to copy a file from a remote host, changing networks, and coming back three days later and be able to resume the transfer.

In terms of speed, it should be competitive with rsync, robocopy, and SMB and native file operations.

We need to support scenarios where blocktransfer is running on the same device as A or B, but also cases where it's not, eg using underyling primitives to transfer blocks over a P2P mesh (for example via the 'iroh' npm package), SMB or plain filesystem primitives.

We should support optional in-flight encryption and transfer of filesystem metadata.

We also want to provide very good feedback in terms of progress of transfer, aiming for at least some kind of accurate reporting every second.

If it's useful, we can also imagine running blocktransfer in more of a client/server scenario - eg running on a server and providing access to resources to blocktransfer clients.

The whole system should be in principle cross-platform in that we could write a Swift or Kotlin native client to do blocktransfers as well.

For now we just want a Node.js based Typescript library that uses Vite and Vitest, along with a Node.js-based CLI client.

We need a safe testing environment that is kept within this folder and not copy any files or risk damaging files on this dev machine. During automated testing, never attempt to use files outside of this repo. But it's OK to access other files as part of the programming and planning phases - as long as its not destructive.