@clinth/blocktransfer
v0.2.2
Published
Verified, resumable filesystem and direct SMB block transfers
Readme
blocktransfer
A portable byte-transfer engine with Node.js filesystem, direct SMB, directory and
CLI adapters. Provider integrations use transferBytes with caller-owned storage,
checkpoints and locking. The existing path/CLI APIs retain adaptive copying into
private staging and full read-back verification by default.
Requires Node.js 22.12 or later. The package is ESM, with TypeScript types and JSDoc included. Peer networking belongs in separate packages; see library boundaries and cross-device integrations. See usage and correctness limits for recovery details and platform constraints, and the implementation checklist for project status.
Opt-in SMB/mounted-volume tests and native macOS performance comparisons accept any explicitly selected volume. Run npm run test:volume -- --volume /Volumes/Temp or npm run bench:volume -- --volume /Volumes/Temp; these create and clean up only uniquely owned test directories. Ordinary tests stay inside the repository.
To compare a mount with direct smb3-client access to the same share, use the
SMB access benchmark, which reports storage I/O separately
from full blocktransfer copies and records cache/security/publication differences.
Library usage
For provider/daemon integrations, use
transferBytes and the store contracts from
@clinth/blocktransfer/core (also exported at the root). This entry has no Node
imports, sidecars, directory planner or subprocesses. Its default verification is
inline; select readback for an independent source/staging pass. The
@clinth/blocktransfer/local entry supplies sidecar-free path/SMB convenience
with caller-owned checkpoints. Existing root path APIs retain their recovery
defaults for compatibility. See the request-by-request status.
Further entries: @clinth/blocktransfer/http (createHttpSource, a range-reading source with Node
connection control; the same function is in /core with an injectable fetch) and
@clinth/blocktransfer/testing (the store contract suites and fault-injection helpers). The root and /local
also export createSourceStore and createSinkStore. See provider-stores.
Import from @clinth/blocktransfer when installed as a dependency. When running examples
inside this checkout, build with npm run build and use ./dist/index.js instead.
The examples assume the input paths already exist and the output files do not.
Paths, file URLs and direct SMB URLs
transferFile, copy and selectEntries accept native paths, URL strings and
URL objects. file:// URLs use the local filesystem; smb:// URLs connect
directly through our ../smb3-client fork, with no mounted share required. The
first SMB path segment names the share. Source and destination can use different
backends, hosts or shares.
await transferFile(new URL('file:///Users/me/video.mp4'),
'smb://user:password@fileserver/temp/video.mp4');
// Omit the password to use a matching system credential.
await copy('smb://user@fileserver/temp/photos', './photos', { recursive: true });
// Applications can supply their own secret-store integration.
await transferFile('./video.mp4', 'smb://fileserver/temp/video.mp4', {
smb: { credentials: async ({ host, share, username }) =>
myCredentialStore.lookup({ host, share, username }) },
});Percent-encode usernames, passwords and individual path segments containing
reserved characters; setting URL.username and URL.password also encodes them.
DOMAIN;user or DOMAIN\user account forms are supported. An explicit password
(including an empty one) takes precedence over credential lookup. Quote URL
arguments in the shell. Inline credentials can appear in shell history/process
arguments; omit them when using a secret store.
System credential lookup (opt in with smb.systemCredentials: true; the CLI enables it unless --no-keychain) uses macOS SMB internet-password entries, Linux
Secret Service via secret-tool (include the username in the URL), or Windows
generic credentials named blocktransfer:smb://host/share (include a nondefault
port when used). Windows domain credentials are not treated as exportable
passwords. The library does not read the keychain by default. Stores may require unlocking. Blocktransfer does not save passwords
to recovery state, progress, results or errors, and it does not create credentials.
See credential-store details.
Direct SMB retains block resumption, independent read-back, conflict checks and
atomic publication. Server-held exclusive handles coordinate competing SMB URL
writers. auto uses block copying; native/ditto/rsync strategies require filesystem
paths. SMB preserves modification times and the read-only attribute, rather than
all POSIX permissions or ACLs. File data is flushed; portable SMB directory fsync
is unavailable. SMB signing is required by default; encryption is off by default; use
smb.encryption: 'if-offered'/'required' (--smb-encryption) to enable SMB3 transport encryption.
Cancellation lets pending SMB requests settle, interrupting stalled connections
after 500 ms. Uncertain remote outcomes retain recovery evidence.
Transfer a file
import { transferFile } from '@clinth/blocktransfer';
const result = await transferFile('./input/video.mp4', './output/video.mp4');
console.log(`Completed ${result.size} bytes in ${result.elapsedMs} ms`);
console.log(`Copied ${result.copiedBytes}; reused ${result.reusedBytes}`);The destination is an exact filename; parent directories are created as needed.
Existing files are refused by default. To intentionally replace one, pass
{ overwrite: true }. Basic permissions and timestamps are preserved; ownership,
ACLs and extended attributes are not transferred.
Copy directory contents or select files
import { copy } from '@clinth/blocktransfer';
// Copy all contents, including nested files and empty directories.
const files = await copy('./input', './output', { recursive: true });
console.log(`Completed ${files.length} files`);
// Copy matching files into another destination, keeping their relative paths.
const selected = await copy('./input', './text-output', {
glob: ['**/*.txt', '**/*.md'],
maxDepth: 3,
});
console.log(`Completed ${selected.length} matching files`);Directory sources copy their contents into the destination. Without selection
options, only immediate files are copied. Globs enable traversal as needed and use
/ separators on every platform; matching dotfiles are included. maxDepth: 0
selects immediate files, 1 includes files in immediate subdirectories, and so on.
Empty directories are preserved when traversing without globs.
copy also accepts a file source and returns a one-element result array. Directory
results contain one entry per file, including unchanged files skipped on resume.
Unrelated destination files are preserved. Directory copies publish one file at a
time and require non-overlapping source/destination trees. Selected symlinks and
special files are unsupported; use a stable source tree rather than expecting a
snapshot of a changing directory.
Preview a selection without copying anything:
import { selectEntries } from '@clinth/blocktransfer';
for await (const entry of selectEntries('./input', { glob: '**/*.txt' })) {
console.log(entry.kind, entry.relativePath);
}Observe progress, cancel and resume
import { transferFile, TransferError } from '@clinth/blocktransfer';
const controller = new AbortController();
// A UI cancel action could call controller.abort(). This example uses a timeout.
const timeout = setTimeout(() => controller.abort(), 5_000);
let paused = false;
try {
await transferFile('./input/archive.bin', './output/archive.bin', {
signal: controller.signal,
onProgress(progress) {
console.log(progress.method, progress.phase, progress.processedBytes, progress.totalBytes);
if (progress.etaSeconds !== null) {
console.log(`Copy pass: about ${Math.ceil(progress.etaSeconds)} seconds left`);
}
},
});
} catch (error) {
if (error instanceof TransferError && error.code === 'ABORTED') {
paused = true;
console.log('Paused; staged data is retained.');
} else {
throw error; // Includes native filesystem errors such as ENOSPC.
}
} finally {
clearTimeout(timeout);
}
if (paused) {
// Resume now or in a later process, using the same paths and a fresh signal.
const result = await transferFile('./input/archive.bin', './output/archive.bin');
console.log(`Reused ${result.reusedBytes} bytes`);
}Callbacks are synchronous and should return quickly. Progress is per file, emitted
at phase changes, block boundaries and approximately every second during pending
asynchronous I/O. processedBytes is the copy/compare pass;
verifiedBytes tracks the separate read-back pass. Copy/reuse counters are logical
bytes, not physical disk I/O. ETA excludes verification and is null outside copying.
For directory totals, supply onDirectoryProgress(progress) to copy. It reports
enumeration, completed/active files, processed/verified bytes, rate and copy ETA.
The CLI displays this aggregate view automatically. Native copiers emit a heartbeat
with bytesKnown: false while byte progress is unavailable.
Every file progress event includes method: pending, block, native, ditto,
rsync, rsync-batch, smb or skipped. Directory events include the active
method (or mixed) and a methods list. Rsync also reports transferredBytes;
these bytes can still be buffered and are separate from completed copying,
verification and publication.
Call controller.abort() to stop. Default cancellation retains resumable staging;
use cleanupOnCancel: true (CLI --cleanup-on-cancel) to discard owned unpublished
data and recovery records. Disposable batch work trees are always removed after
the copier closes. Already published files and uncertain publication evidence are
retained. Empty coordination directories may remain. See
cancellation limits.
Recovery and verification are independently configurable with { resumable: false }
and { verify: false }, or CLI --non-resumable and --non-verified. Both remain
enabled by default. engine: 'auto' (the default; strategy is the older name) learns from
completed files in the current process. You can name an engine instead: block, fs, cp,
cpy (optional npm module), ditto (macOS), rsync, or smb2 (the smb2 command line tool,
direct SMB). listEngines() / --list-engines shows what works on your machine; see
Engines.
Library users get onEvent lifecycle events (started, progress, then one of finished, paused
or error), hidden recovery markers at the destination and source, and findPendingOperations,
probeOperation, resumeOperation and cancelOperation to pick up unfinished work at startup; see
Lifecycle events and pending operations.
See mode semantics and adaptive limits.
The backend investigation compares rsync with equivalent guarantees and assesses direct SMB publication.
On macOS/Linux, automatic directory copying can batch eligible small files through
rsync, then compare completed batch timings with the built-in engine. Both paths
retain the requested recovery and verification guarantees. Rsync is optional:
missing/incompatible executables fall back to built-in copying. Use
allowRsync: false / --no-rsync to prohibit it, or rsyncPath / --rsync-path
to select an executable. Windows automatic copying uses the portable engine.
The cross-device investigation records locality and transport
experiments that can inform a separate networking package.
Resume directory operations with the same paths and selection options. Source
changes pause by default; { follow: true, followRetryMs: 250 } retries until the
source settles or the signal is cancelled, then returns after a successful copy.
Destination changes after starting are a separate conflict: overwrite alone does
not discard them. Only pass overwriteDestinationChanges: true after reviewing
those edits and deciding to replace them.
API reference
All exports below are available from blocktransfer; types are TypeScript-only.
| Export | Purpose |
| --- | --- |
| transferFile(source, destination, options?) | Transfer one file; returns Promise<TransferResult>. |
| copy(source, destination, options?) | Transfer a file or directory contents; returns Promise<TransferResult[]>. |
| selectEntries(root, options?) | Preview selected entries as an async iterator of SelectedEntry. |
| stateDirectory(destination) | Compute the absolute local path or credential-free SMB URL for recovery, without I/O. |
| TransferError | Library error with a machine-readable code; native I/O errors also propagate. |
| TransferOptions, SelectionOptions, CopyOptions | Transfer controls, selection rules, and their combination. |
| TransferLocation, SmbOptions, SmbCredentials, SmbCredentialRequest, SmbCredentialProvider, systemSmbCredentials(request) | Resource inputs, direct SMB policy and system/custom credential resolution. |
| Progress, DirectoryProgress, TransferPhase, TransferMethod, TransferResult, SelectedEntry, TransferErrorCode, PublicationAdapter | Typed progress, copier identification, results, selections, failures and an explicit atomic publication capability. |
blockSize defaults to 1 MiB (maximum 64 MiB); checkpointBlocks defaults to 16.
Resume reuses the saved block size; explicitly supplying a different size is an
error. Recovery directories live beside destinations and may remain empty after
success. Do not edit recovery data while a transfer is active.
Verification requires an unchanged observed source; it cannot provide a coherent snapshot of an arbitrarily changing file. Publication and crash durability depend on filesystem/OS support. Read the platform limits before relying on the library for concurrent writers or power-loss guarantees.
Library boundaries and cross-device integrations
Blocktransfer owns transfer correctness and efficient work against storage: selection, hashing, block reuse, staging, recovery, conflict detection, verification, publication and progress. SMB is a storage backend, using the local fork for its protocol, authentication and connection security. Peering, client/server services, discovery and peer transfer protocols belong in another package. That package could use WebSockets, iroh, HTTP or another data backend without making any of them a requirement for local copying.
The intended design lets that package invoke blocktransfer beside each resource. Source hashing stays beside the source; destination comparison, staging and read-back verification stay beside the destination. Only manifests, missing data and status need to cross the network. Merely putting a remote filesystem API under the current copy loop would retain expensive remote reads and round trips.
Available today
The supported entry points are the API reference above. Native
paths and file:// URLs access the calling process's filesystem; smb:// URLs
access SMB storage directly. The built-in SMB adapter still reads verification
bytes across SMB in the calling process.
transferBytes, SourceStore, SinkStore, Checkpoint and StateStore are public
provider integration APIs. A separate package can implement source/sink operations
beside its storage, using copyRange, source.digest and sink.digest to keep
payload processing in its provider process. Checkpoints and locks are caller-owned;
the core imposes no path or transport. See the
provider contract. This supplies a single-item engine,
not an exported peer job protocol or manifest scheduler.
TransferOptions.publicationAdapter is an existing, narrow integration hook. Its
renameExclusive(staging, destination) operation must atomically publish the same
staged object, preserve its identity/metadata, refuse an existing destination with
EEXIST, and wait until the mounted filesystem observes the rename. An adapter
must establish that both paths belong to the same storage namespace. Errors
propagate without a fallback because publication may already have happened.
This hook applies to new destinations; replacements use filesystem publication.
It is useful for storage-specific optimizations, not a general source, sink or
transport interface. Credentials and callback state are not saved in recovery data.
Intended extension points
The broader paged-session/planner boundaries below remain a design, beyond the public single-item store interfaces above. Internal legacy manifests, filesystem helpers and directory journals are not supported integration APIs.
| Component | Responsibility in blocktransfer | | --- | --- | | Source session | Enumerate selected entries, observe source identity/generation, compute paged hash manifests, read requested ranges and revalidate the source before completion. An observed generation does not imply a filesystem snapshot. | | Block planner | Compare source digests with locally validated destination/staging digests and return bounded pages of missing ranges. Keep this computation independent of data delivery. Start with the existing fixed-offset blocks and SHA-256; shifted-block matching is separate work. | | Destination session | Own locks, private staging, conflict policy and recovery state; accept validated blocks, checkpoint, independently read back storage, and publish with a recoverable completion receipt. | | Storage capability adapter | Supply local positioned I/O, identity observations, staging, flush, metadata and publication operations. Explicitly describe capabilities so a filesystem or future embedded-device backend cannot silently weaken the requested guarantees. | | Progress and cancellation | Expose bounded asynchronous work and phase events for hashing, comparison, writing, verification and publication. Preserve cooperative cancellation and the default resumable, verified behavior. |
The separate integration package owns endpoint lifecycle, path/export authorization, authenticated manifest exchange, protocol versioning, transport framing, retries, connection changes and data scheduling. It translates core job operations and events into its own protocol. Blocktransfer should not require a socket, peer identity or connection ID to open or resume a storage job. Transport frame sizes and stream counts must remain independent of content block sizes.
A cross-device integration would use these boundaries as follows:
- Open a source session and send its manifest pages, binding file identities, sizes, block layout, digests and the observed source generation to the job.
- Open a destination session with an explicit conflict policy. Hash existing or staged data locally and return the missing ranges; saved checkpoints alone are not proof that bytes still match.
- Deliver only the needed blocks through the chosen data backend. Validate each block's job/generation, range, length and digest. Bound outstanding bytes and let destination storage pressure control backpressure.
- Revalidate the source locally and independently read back and hash destination staging locally against the bound manifest. Hashing received buffers alone does not replace filesystem read-back. Changed sources or destinations pause the job unless an explicit policy permits reconciliation or replacement.
- Publish through the destination session and obtain its completion receipt. Distinguish received, written, durably checkpointed, verified and published states. After a lost reply, query/reconcile the job instead of blindly repeating publication. A successful socket send is not a durable checkpoint.
This can reduce network traffic without removing verification: a fresh file need not be sent back to its source for read-back hashing, and a resumed transfer need only send missing blocks. Both endpoints still perform local reads. Actual speed depends on disk, hashing CPU, transport and workload; the locality experiments are loopback research, not a cross-device throughput guarantee. Random sampling may detect differences early, but cannot prove equality or replace full verification by default.
Extraction should preserve the existing local API and its recovery/conflict
tests, then expose independently testable source, planner and destination
operations. A separate package can build networking around those operations;
this package's CLI remains a filesystem/SMB copier. The same storage
jobs should remain usable without a network stack, with explicit
resumable: false and verify: false policies available when requested.
CLI and development
npm ci
npm run check
node dist/cli.js --help
node dist/cli.js ./input/video.mp4 ./output/video.mp4
node dist/cli.js --glob '**/*.txt' ./input ./outputCtrl+C pauses a transfer; run the same command to resume. Quote globs so the shell does not expand them. See the usage guide for more CLI options, conflict handling, tests and benchmarks.
Compare copying performance
npm run bench:compare -- --mib 256 --files 1000 --repeats 3
# Or select a mounted destination:
npm run bench:compare -- --volume /Volumes/Temp --mib 64 --files 200Writes a Markdown report and raw JSON under benchmarks/results/, comparing
blocktransfer's four resume/verification modes against cp, ditto and rsync on
small files, a large file and partial-file resume. Native verified variants include
the time to launch a command and freshly hash both source and destination.
An additional correctness audit is reported separately. See
benchmark options and timing semantics.
Original project brief
Historical context follows. The package boundary above supersedes suggestions below to include peering or client/server services in blocktransfer itself.
This is a library (and separate CLI) to do block-based transfer of data within and between filesystems and devices.
The emphasis is on speed, reliability and accuracy.
At a basic level it should support transfering a file from point A to B. If anything goes wrong, it should support resumability, guaranteeing that the transfer is byte-perfect and with minimal amount of repeated data transfer. Similar to what rsync can offer. For example, starting to copy a file from a remote host, changing networks, and coming back three days later and be able to resume the transfer.
In terms of speed, it should be competitive with rsync, robocopy, and SMB and native file operations.
We need to support scenarios where blocktransfer is running on the same device as A or B, but also cases where it's not, eg using underyling primitives to transfer blocks over a P2P mesh (for example via the 'iroh' npm package), SMB or plain filesystem primitives.
We should support optional in-flight encryption and transfer of filesystem metadata.
We also want to provide very good feedback in terms of progress of transfer, aiming for at least some kind of accurate reporting every second.
If it's useful, we can also imagine running blocktransfer in more of a client/server scenario - eg running on a server and providing access to resources to blocktransfer clients.
The whole system should be in principle cross-platform in that we could write a Swift or Kotlin native client to do blocktransfers as well.
For now we just want a Node.js based Typescript library that uses Vite and Vitest, along with a Node.js-based CLI client.
We need a safe testing environment that is kept within this folder and not copy any files or risk damaging files on this dev machine. During automated testing, never attempt to use files outside of this repo. But it's OK to access other files as part of the programming and planning phases - as long as its not destructive.
