Simple snowball sync

Sync folders of data onto a snowball

Features:

• Start and stop the process at any stage
• Compute SRI sha256 hash of all files and stored inside of the snowball in a manifiest.json
• Compress small files into a tar.gz so snowball can batch extract when importing into s3
• Validate uploaded files against the computed hash

Manifest

Create a list of files to be synced

sss manifest /media/$USER/external-drive

Sync

Start the syncing processes

sss sync manifest.json --endpoint 10.254.32.104 --target s3://linz-snowball-a/drive1

Syncing can be stopped and started at any time

Options:

• --filter Files under this limit will be compressed into a .tar.gz and uploaded using the snowball batching (Mb) --filter 1 only directly upload files > 1MB
• --concurrency Number of files to upload at a single time
• --verbose More verbose logging