WebDAV Bulk Transfer

Introduction

This tool is designed to bulk download and upload files through WebDAV. This tool can also transfer files from a WebDAV server to another (mixing download and upload methods) in one command. By the way, the tool is designed to

• handle files recursively (recursive option)
• upload only files that does not exists on the target server (onlynewfiles option)

Dependencies

This tool uses webdav under the hood to connect to the webDAV server when using download or transfer methods to fetch the source server (act as a fs.readdir to get directories and files in a given directory path).

Installation

• Go to the directory where you cloned the repository.
• Run npm install @jordanebachelet/webdav-bulk-transfer.

Authentication

As this tool uses webdav under the hood to connect to the webDAV server, it will allow you to use the following authentication methods:

• No authentication (don't provide any username and password)
• Username and password through Basic authentication
• OAuth token authentication
  • Through the Command Line interface, you can provide the OAuth token object as a stringified string through the username parameter
  • Throught the Javascript interface, you can directly provide the OAuth token object through the username parameter (the object as is)
• If the source/target server is using two-factor authentication (username/password and certificates), you can provide a certificate path and a passphrase for the authentication

How to use

Through command line interface

You can run the npm link command before using this tool. This will allows you to directly run the davtransfer command in your command line interface.

Then, you can run the following commands:

Download

// with node command
node bin/cli.js download "https://server.url" "path/to/local/directory" "/path/to/remote/directory,/path/to/remote/file.ext" --username "username" --password "password" --recursive --concurrency 10 --keeplocaldirectory
// after a "npm link"
davtransfer download "https://server.url" "path/to/local/directory" "/path/to/remote/directory,/path/to/remote/file.ext" --username "username" --password "password" --recursive --concurrency 10 --keeplocaldirectory

This commands will download directories and files recursively from the given server to the given local directory (10 concurrent connections will be opened each time)

Upload

// with node command
node bin/cli.js upload "path/to/local/directory" "https://server.url" --username "username" --password "password" --recursive --concurrency 10 --cleanafter
// after a "npm link"
davtransfer upload "path/to/local/directory" "https://server.url" --username "username" --password "password" --recursive --concurrency 10 --cleanafter

This commands will upload files in the local directory recursively to the given server (10 concurrent connections will be opened each time). Then the local directory will be removed.

Transfer

// with node command
node bin/cli.js transfer "https://source.server" "https://target.server" --sourceusername "username" --sourcepassword "password" --targetusername "username" --targetpassword "password" --recursive --concurrency 10 --cleanafter --keeplocaldirectory
// after a "npm link"
davtransfer transfer "https://source.server" "https://target.server" --sourceusername "username" --sourcepassword "password" --targetusername "username" --targetpassword "password" --recursive --concurrency 10 --cleanafter --onlynewfiles --keeplocaldirectory

This commands will transfer files recursively from the given source server to the given server (10 concurrent connections will be opened each time). Then the local directory will be removed.

Through the Javascript interface

You can use this tool directly in your NodeJS scripts.

All methods returns a native Promise so you can do some work after the method execution.

Hereafter how to integrate this tool in your scripts:

Download

const webdavBulkTransfer = require('webdav-bulk-transfer');

webdavBulkTransfer.download('https://server.url', 'path/to/local/directory', ['/path/to/remote/directory','/path/to/remote/file.ext'], {
    username: 'username',
    password: 'password',
    recursive: true,
    concurrency: 10,
    keeplocaldirectory: true
}).then(downloadedFiles => {
    console.log(`${downloadedFiles.length} files downloaded.`);
});

This command will download directories and files recursively from the given server to the given local directory (10 concurrent connections will be opened each time)

Upload

const webdavBulkTransfer = require('webdav-bulk-transfer');

webdavBulkTransfer.upload('path/to/local/directory', 'https://server.url', {
    username: 'username',
    password: 'password',
    recursive: true,
    concurrency: 10,
    cleanafter: true,
    keeplocaldirectory: true
}).then(uploadedFiles => {
    console.log(`${uploadedFiles.length} files uploaded.`);
});

This command will upload files in the local directory recursively to the given server (10 concurrent connections will be opened each time). Then the local directory will be removed.

Transfer

const webdavBulkTransfer = require('webdav-bulk-transfer');

webdavBulkTransfer.transfer('https://source.server', 'https://target.server', ['/path/to/remote/directory','/path/to/remote/file.ext'], {
    sourceusername: 'username',
    sourcepassword: 'password',
    targetusername: 'username',
    targetpassword: 'password',
    recursive: true,
    concurrency: 10,
    cleanafter: true,
    onlynewfiles: true,
    keeplocaldirectory: true
}).then(transferedFiles => {
    console.log(`${transferedFiles.length} files transfered.`);
});

This command will transfer files recursively from the given source server to the given server (10 concurrent connections will be opened each time). Then the local directory will be removed.

API Reference

Please check the docs folder here.