swarm-deploy
v0.1.0
Published
Secure resumable artifact uploads over direct HyperDHT
Readme
Swarm Deploy moves build artifacts from authorized clients to one receiving server. Each file uses a fresh direct HyperDHT connection authenticated and encrypted with Noise. The client pins the server's public key, while the server accepts only client public keys in its immutable startup allowlist.
The receiver is upload-only. It does not execute, serve, or provide a download protocol for stored artifacts.
Requirements
- Node.js 22 or 24, or the current stable Bare runtime.
- A persistent server seed and at least one independently generated client seed.
- A dedicated server storage directory.
- Network access suitable for HyperDHT.
Install
Install the runtime API in an application:
npm install swarm-deployInstall the CLI globally for a receiving server or CI uploader:
npm install --global swarm-deploy
swarm-deploy --helpSecurity and identity model
Three values have different roles:
- Seed: private 32-byte identity material. Prefer a protected seed file or
environment secret. The CLI also accepts
--seed <64-lower-hex>, but command arguments can be exposed through shell history, process listings, and CI tracing. Never put a seed in an allowlist or application log. - Client public key: derived from a client seed and installed in the server's allowlist.
- Server public key: derived from the server seed and pinned by every client. It is both the direct HyperDHT destination and the server identity commitment.
HyperDHT Noise authenticates both peers and encrypts the transport. Application SHA-256 checks verify the deterministic TAR and extracted file bytes.
There is no topic, swarm discovery, Protomux channel, dynamic allowlist reload, or reconnect budget. Changing an allowed client key requires a controlled server restart.
Keep seeds stable to preserve identity. Copying one client seed to several machines intentionally gives all of them the same uploader identity; use separate client seeds when independent authorization is required.
Provision identities
Generate separate server and client seed files:
swarm-deploy keygen --out server.seed
swarm-deploy keygen --out client.seedkeygen creates an owner-only file, refuses to overwrite an existing path, and
prints the corresponding public key—not the seed.
Recover public keys later:
SERVER_KEY=$(swarm-deploy public-key --seed-file server.seed)
CLIENT_KEY=$(swarm-deploy public-key --seed-file client.seed)Public keys are lowercase 64-character hexadecimal strings and are safe to use as configuration values.
Quick start
Start the receiver:
swarm-deploy server --seed-file server.seed --storage /srv/artifacts \
--allow-key "$CLIENT_KEY" --max-file-bytes 1073741824 --max-staging-bytes 2147483648The server prints its full public key and then ready after storage recovery,
scrub, retention initialization, and HyperDHT listening complete:
0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
readyUpload a file:
swarm-deploy upload --seed-file client.seed --server-key "$SERVER_KEY" \
--idle-timeout 60000 ./artifact.binSuccessful output is:
artifact.bin COMMITTEDUploading the same managed content again returns ALREADY_COMMITTED without
retransmitting its TAR bytes.
CLI reference
keygen
swarm-deploy keygen --out <seed-file>Creates a new seed file with owner-only permissions and prints its public key. The destination must not already exist.
public-key
swarm-deploy public-key --seed-file <seed-file>
swarm-deploy public-key --seed <64-lower-hex>Reads a seed file or canonical seed string and prints its public key.
server
swarm-deploy server \
--seed-file <seed-file> \
--storage <directory> \
--allow-key <64-lower-hex> \
--max-file-bytes <bytes> \
--max-staging-bytes <bytes> \
[--allow-key <64-lower-hex>]... \
[--max-storage-bytes <bytes>] \
[--max-age-days <days>] \
[--replace-name <safe-basename>]...Replace --seed-file <seed-file> with --seed <64-lower-hex> to provide the
seed inline.
Required options:
- Exactly one seed source:
--seed-file,--seed, orSWARM_DEPLOY_SERVER_SEED. --storage: dedicated artifact and internal-state root.--allow-key: authorized client public key. Repeat for multiple identities; duplicates and malformed keys are rejected.--max-file-bytes: maximum extracted artifact size.--max-staging-bytes: aggregate persistent staging reservation. An admitted transfer reserves its deterministic TAR size plus extracted file size, so this commonly needs to be at least twice the largest simultaneously staged payload.
Optional options:
--max-storage-bytes: maximum total managed committed storage. Oldest eligible artifacts are removed first.--max-age-days: remove eligible committed artifacts at or beyond this age.--replace-name: permit replacement of this exact basename. Repeat for multiple mutable names.
The CLI requires at least one --allow-key. Its snapshot is immutable for the
life of the process.
upload
swarm-deploy upload \
--seed-file <seed-file> \
--server-key <64-lower-hex> \
[--idle-timeout <milliseconds>] \
<file-or-directory>Replace --seed-file <seed-file> with --seed <64-lower-hex> to provide the
seed inline.
--server-keyis the full pinned server public key.--idle-timeoutdefaults to 60 seconds and bounds inactive protocol reads and backpressured writes.- The input may be any regular binary file; Swarm Deploy creates the canonical one-entry USTAR stream automatically. Pre-tarring is not required.
- A direct file retains its basename.
- A directory processes immediate regular-file children once, in lexical order, with one independent connection and result per file.
- Subdirectories, symlinks, non-regular files, unsafe names, and names in the
reserved
history-namespace are skipped and reported. - An unreadable directory entry is reported as a failure while later entries continue.
Accepted names start with an ASCII letter or digit, contain only letters,
digits, ., _, and -, and occupy at most 100 UTF-8 bytes. history- is
reserved for server-managed replacement history.
If a basename is exactly 64 lowercase hexadecimal characters, pass it with a
directory component such as ./<name> so the CLI does not treat it as an
accidentally pasted seed.
Seed sources
Server and upload commands accept exactly one of:
--seed-file <seed-file>--seed <64-lower-hex>- the role-specific
SWARM_DEPLOY_SERVER_SEEDorSWARM_DEPLOY_CLIENT_SEEDenvironment variable
String values must contain exactly 64 lowercase hexadecimal characters. Combining seed sources is an error.
The public-key command accepts --seed-file or --seed; it does not consume a
role-specific environment variable.
Prefer seed files or protected environment variables in production. Use
--seed only when exposure through command history, process inspection, and
tooling logs is acceptable.
CLI exit codes
0: every selected file was committed or already committed.1: upload, network, protocol, storage, cleanup, or runtime failure.2: usage or configuration error.
For a directory, the CLI prints one line for every selected or skipped entry and
returns 1 if any selected upload failed.
CI uploader example
Store the client seed as a protected CI secret and the server public key as a nonsecret variable:
- name: Install uploader
run: npm install --global swarm-deploy
- name: Upload artifact
env:
SWARM_DEPLOY_CLIENT_SEED: ${{ secrets.SWARM_DEPLOY_CLIENT_SEED }}
SWARM_DEPLOY_SERVER_KEY: ${{ vars.SWARM_DEPLOY_SERVER_KEY }}
run: |
swarm-deploy upload \
--server-key "$SWARM_DEPLOY_SERVER_KEY" \
--idle-timeout 60000 \
./dist/artifact-linux-x64.tar.gzDo not enable shell tracing around commands that read seed environment
variables or pass --seed.
Transfer and resume behavior
Each file follows this lifecycle:
- The client connects directly to the pinned server public key with its seeded HyperDHT identity.
- The server firewall and connection handler verify the authenticated client key against the startup allowlist.
- The client sends bounded metadata containing the name, file size and digest, deterministic TAR size and digest, and transfer ID.
- The server responds with
ACCEPT,RESUME,VERIFIED,ALREADY_COMMITTED, or a stable rejection. - The client sends exactly the required deterministic one-entry USTAR bytes.
- The server validates the canonical archive, extracted size, and both SHA-256 values before durable commit.
- Success requires an explicit terminal
COMMITTEDresult. EOF or socket closure is never success.
Incomplete uploads retain only durable TAR progress. On reconnect, the server returns a TAR-byte offset and SHA-256 of that prefix. The client regenerates and compares the prefix before sending the suffix. A mismatch closes that connection and retries once from zero with explicit reset intent.
The server coalesces network fragments into 1 MiB durability batches plus the final remainder. A disconnect can require retransmitting only the uncheckpointed in-memory tail; it never advertises bytes that were not durably published.
Inactive sessions expire after seven days by default. Active receives and verification are protected from expiry.
Storage, replacement, and retention
The storage root contains visible current and historical artifacts plus the
reserved .swarm-deploy/ internal directory. Do not modify that directory
while the server is running.
Names are create-only by default:
- New content for an unused name is committed atomically.
- Identical managed content returns
ALREADY_COMMITTED. - Different content for an occupied create-only name returns
FILE_EXISTS. - Unmanaged files and paths are never overwritten or deleted.
Names configured with --replace-name or ServerOptions.replaceNames are
mutable:
- Different verified content atomically becomes current.
- The prior managed inode and record are preserved as
history-<full-old-transfer-id>. - The current mutable name is pinned against age and quota retention.
- Historical versions remain eligible for retention.
Commit journals and inode checks recover interrupted create and replacement operations. Recovery rolls back mutations before the durable new current sidecar and rolls forward operations after that linearization point.
Optional retention applies to managed artifacts only:
maxAge/--max-age-daysremoves eligible artifacts by age.maxStorageBytes/--max-storage-bytesremoves the oldest eligible artifacts until under quota.- Startup recovery re-hashes managed files. Scheduled cleanup validates managed metadata and file sizes, removes invalid managed records safely, reports unknown paths without deleting them, and applies age and quota retention.
Runtime defaults:
- 64 authenticated connections.
- 8 active uploads.
- 60-second upload inactivity timeout.
- 1 GiB minimum free-disk reserve.
- 15-minute cleanup interval.
- 7-day resumable-session lifetime.
The advanced runtime API can override these values; the server CLI intentionally exposes only its required limits, committed retention, and replacement policy.
Runtime API
The package is strict TypeScript and exposes the same root API to ESM and CommonJS consumers.
Identity helpers
import {
generateSeed,
keyPairFromSeed,
parsePublicKey,
parseSeed,
publicKeyFromSeed
} from 'swarm-deploy'
const seed = generateSeed() // 32 random bytes
const restored = parseSeed(process.env.SEED!) // strict lowercase hex
const keyPair = keyPairFromSeed(restored)
const publicKey = publicKeyFromSeed(restored)
const peer = parsePublicKey(process.env.PEER_PUBLIC_KEY!)parseSeed and parsePublicKey require exactly 64 lowercase hexadecimal
characters. keyPairFromSeed is deterministic.
parseAllowlist(text) parses lowercase client public keys separated by
newlines. Blank lines and lines beginning with # are ignored; malformed or
duplicate keys throw. It returns a Set<string> suitable for
ServerOptions.allowedKeys.
Server
import { Server, parsePublicKey } from 'swarm-deploy'
const server = new Server({
seed: process.env.SWARM_DEPLOY_SERVER_SEED!,
storageDir: '/srv/artifacts',
allowedKeys: [
parsePublicKey(process.env.CLIENT_A_PUBLIC_KEY!),
parsePublicKey(process.env.CLIENT_B_PUBLIC_KEY!)
],
maxFileBytes: 1024 ** 3,
maxStagingBytes: 4 * 1024 ** 3,
maxConnections: 64,
maxActiveUploads: 8,
idleTimeout: 60_000,
cleanupInterval: 15 * 60_000,
resumeTtl: 7 * 24 * 60 * 60_000,
minFreeBytes: 1024 ** 3,
maxAge: 15 * 24 * 60 * 60_000,
maxStorageBytes: 100 * 1024 ** 3,
replaceNames: ['release.tar.gz', 'latest.json']
})
await server.listen()
console.log(server.publicKey.toString('hex'))
// Later, after draining or on process shutdown:
await server.close()Required ServerOptions:
seed: Buffer | stringstorageDir: stringallowedKeys: Iterable<Buffer | string>maxFileBytes: numbermaxStagingBytes: number
Optional operational limits and policies:
maxConnections,maxActiveUploads,idleTimeoutcleanupInterval,resumeTtl,minFreeBytesmaxAge,maxStorageBytes,replaceNames
Advanced integration and test seams:
dhtordhtFactoryfor an injected HyperDHT node.storagefor a compatible filesystem adapter.schedulerfor timeout and interval control.loggerwith optionalinfo,warn, anderrormethods.
Client
import { Client, parsePublicKey } from 'swarm-deploy'
const client = new Client({
seed: process.env.SWARM_DEPLOY_CLIENT_SEED!,
serverPublicKey: parsePublicKey(process.env.SWARM_DEPLOY_SERVER_KEY!),
connectTimeout: 30_000,
idleTimeout: 60_000
})
const result = await client.upload('./dist/release.tar.gz')
console.log(result.status, result.name, result.size)
await client.close()ClientOptions.seed accepts a Buffer or canonical lowercase 64-character hex
string. serverPublicKey is a Buffer produced by parsePublicKey. Optional
values are connectTimeout, idleTimeout, dht, dhtFactory, and logger.
connectTimeout defaults to 30 seconds. idleTimeout defaults to 60 seconds.
Calling close() aborts pending work, closes active sockets, and is idempotent.
Upload results
A direct file resolves to:
interface UploadResult {
status: 'COMMITTED' | 'ALREADY_COMMITTED'
name: string
size: number
digest: Buffer
transferId: Buffer
}A directory resolves to:
interface BatchUploadResult {
status: 'COMMITTED' | 'FAILED'
results: Array<
| UploadResult
| {
name: string
status: ErrorCode
reason?: string
}
>
skipped: Array<{
name: string
path: string
reason: SkippedUploadReason
}>
}Directory members are processed sequentially. A failed member does not prevent later members from being attempted.
Events
Server and Client are event emitters. Listener and logger exceptions are
contained and cannot change protocol correctness.
Public peer and transfer correlation fields use 12-character SHA-256 fingerprints. Events never expose seeds, secret keys, full remote public keys, TAR contents, or resumable session material. Internal storage warnings may include a full SHA-256 uploader fingerprint, but never the uploader key itself.
Server events:
authentication: accepted client fingerprint.connection,connection-open,connection-close: authenticated connection lifecycle and current count.offer: accepted, resumed, reset, rejected, or already-committed state.progress: durable TAR bytes received and total TAR bytes.verification: started, succeeded, or failed.commit: succeeded or failed.recovery: startup, per-journal, corruption, resumable, and completion outcomes.retention: startup, scheduled, manual, commit, or post-commit outcomes.failure: stable failure code and peer fingerprint.listening: local server-key fingerprint.close: closed or failed outcome.
Client events:
connection,connection-open,connection-closeoffer: offered, accepted, resumed, reset, rejected, or already committed.progress: cumulative TARbytesSentand fulltotalBytes, including a durable resume offset.verification,commitresult: direct result or per-file/aggregate directory result.skipped: skipped directory member and stable reason.failure,close
Use the exported ServerEventMap, ClientEventMap, ServerEventName, and
ClientEventName types for event-name-specific payload narrowing:
client.on('progress', ({ name, bytesSent, totalBytes }) => {
console.log(name, `${bytesSent}/${totalBytes}`)
})
server.on('recovery', (event) => {
if (event.status === 'failed') console.error(event.phase, event.reason)
})Logging
Both constructors accept:
interface Logger {
info?(message: string, details?: Record<string, unknown>): void
warn?(message: string, details?: Record<string, unknown>): void
error?(message: string, details?: Record<string, unknown>): void
}Logger exceptions are ignored. Logger detail objects are diagnostic rather than
a stable ingestion schema; use typed event payloads or SwarmDeployError.code
for automation. Do not use fingerprints as credentials.
Errors
Configuration, authentication, protocol, transfer, and managed-storage failures
generally reject with SwarmDeployError. Raw operating-system or adapter errors
may propagate while selecting or opening a local input, initializing or locking
the server storage root, or performing filesystem operations:
import { ERRORS, SwarmDeployError } from 'swarm-deploy'
try {
await client.upload('./artifact.tgz')
} catch (error) {
if (error instanceof SwarmDeployError) {
console.error(error.code, error.message)
if (error.code === ERRORS.CONNECT_TIMEOUT) {
// Server could not be reached and authenticated before the deadline.
}
} else {
console.error(error)
}
}Stable codes include authentication and server-key rejection, invalid
configuration and protocol records, file and staging limits, disk reserve,
filename and replacement conflicts, checksum failures, connection or upload
timeouts, aborts, commit failures, and cleanup failures. Import ERRORS rather
than matching exception messages.
Production operations
- Run the server under a dedicated non-root account.
- Restrict the server seed and storage root to that account.
- Supervise the process and wait for the final
readyline before marking it healthy. - Restart with the same seed, allowlist, limits, replacement names, and storage root so interrupted sessions and commit journals can recover.
- Alert on nonzero CLI exits and failed authentication, recovery, verification, commit, retention, and cleanup events.
- Never edit
.swarm-deploy/while the server is running. - Stop the server cleanly before backing up or restoring the complete storage root.
- Publish stored artifacts through a separately configured artifact service or web server.
- Roll out an exact package version to a canary before wider deployment.
See SECURITY.md for vulnerability reporting and operator precautions.
Contributor development
Production and tests are strict TypeScript. Build output is generated under
untracked dist/ and .test-dist/. Tests generate binary payloads in temporary
directories and use an isolated local HyperDHT testnet, not the public DHT.
npm ci
npm run build
npm run build:test
npm run test:types
npm run format:check
npm run lint
npm run test:node
npm run test:bare
npm run test:property
npm run test:package
npm run test:release-tagExercise the built CLI directly:
node dist/bin/swarm-deploy.js --help
bare dist/bin/swarm-deploy.js --helpRelease
Version tags use v<package-version>. The tag workflow validates the version,
builds the untracked distribution, checks package contents and types, and
publishes through the configured npm environment with provenance.
See RELEASING.md for the release and rollback procedure.
Protocol specification
See docs/spec/swarm-deploy.md for the complete transport, deterministic TAR, resumability, storage, replacement, recovery, retention, threat-model, and package requirements.
