Datashuttle

Mass file mover for disk-based and S3 file systems. (Experimental)

Installation

npm install -g datashuttle

Command line interface

Currently datashuttle is available as a CLI. API is a future possibility.

1.Configuration file

In order to use the cli create configuration.json in your working folder. This is an example:

{
  "fileSystems": {
    "source":{
      "type":"disk",
      "folder":"C:\\projects\\sl\\data-dump"
    },
    "target":{
      "type":"s3",
      "url":"s3-api.us-geo.objectstorage.softlayer.net",
      "accessKey":"<<your access key>>",
      "secretKey":"<<your secret key>>",
      "buckets": "*"
    }
  },
  "rules":{
    "ignore":["^\\."]
    "maxCores":8,
    "validateMD5":true,
    "allowRestart":true    
  },
  "report":{
    "groups":{
      "small":"file.size<50",
      "medium":"file.size<5000000",
      "large":"*"
    }
  }
}

configuration.json specifies the source and the target file systems for datashuttle. Files in the source will be inventoried into a manifest and/or copied into the target file system.

type

Type of the file system. Currenty there are only two supported types: 's3' and 'disk'. For 'disk' specify the folder where the payload is located, for 'S3' specify the accounts URL and credentials.

1. Exporting files

Use the export operation to move files between file systems of different or same type. Manifest is created in due process.

datashuttle -e export

The export operation will take advantage of multiple cores.

2. Creating a Manifest

Manifest file is an iventory of all the files recursively found in the source file system. The following command creates a manifest.

datashuttle -e manifest

Manifest consists of two files: manifest.json with folder and summary information and manifest.csv enumerating all files found in the source. Creating a manifest is a required step before running a copy operation.

3. Copying files

This command is for copying files from the source to the target file system. Manifest is required.

datashuttle -e copy

4. Rules

ignore

Expression (regular) to exclude file from manifest and copy operations.

maxCores

Maximum number of cores to use. Defaults to the number of cores available.

validateMD5

When enabled creates MD5 hashes for each file in the manifest and validates copied files against these. Not (yet) available for files larger than 50MB when using S3 file systems. Uses up CPU and slows down copy and export processes.

allowRestart

Allows the copy operation to restart skipping the files that have been copied already

5. Report

Use report to include summary information into the manifest. Summary information consists of count of files in each group and their total size. The expession for grouping is javascript string operating on object of type File:

{
  name,
  size
}

(size is expressed in kilobytes)