lvr_bulk_downloader
v0.5.8
Published
Bulk downloader sync or async, retring configuration and flat or tree directories output.
Downloads
14
Maintainers
Readme
LVR Bulk Downloader
Easy downloading...
Tested with 17000 files in input, about 17GB downloaded without an error.
Features
Single file or bulk download;
Retry download if failed;
Output directory flat or tree directories;
Sync or Async download;
Output response for all files downloaded;
Installation
npm i lvr_bulk_downloader --save
Import
import { Downloader, DownloaderTypes } from 'lvr_bulk_downloader';
Simple bulk downalod use
const files = [
new DownloaderTypes.InputFile('http://host.domain/dir/file.ext'),
new DownloaderTypes.InputFile('http://host.domain/dir/file2.ext')
];
const downloader = new Downloader();
downloader.run(files);
Complex bulk download use
const files = [
new DownloaderTypes.InputFile('http://host.domain/dir/file.ext', './download/file.ext'),
new DownloaderTypes.InputFile('http://host.domain/dir/file2.ext', './download/dir/file2.ext'),
new DownloaderTypes.InputFile('http://host.domain/dir/file3.ext', './download/dir/file3.ext')
];
const options = new DownloaderTypes.Options();
options.output_directory = './download/';
options.debug_mode = DownloaderTypes.DebugMode.DEBUG;
options.retry_times = 3;
const downloader = new Downloader(options);
downloader.run(files, function(output: DownloaderTypes.OutputFile[]){
const downloadedCount = output.filter(x=>x.response?.status == DownloaderTypes.Status.OK).length;
const errorCount = output.filter(x=>x.response?.status == DownloaderTypes.Status.KO).length;
});
For sync download adding await to run function
const files = [
new DownloaderTypes.InputFile('http://host.domain/dir/file.ext'),
new DownloaderTypes.InputFile('http://host.domain/dir/file2.ext')
];
const downloader = new Downloader();
await downloader.run(files);
Single file download use
var file = new DownloaderTypes.InputFile('http://host.domain/dir/file.ext');
var downloader = new Downloader();
downloader.run(file);
Types definition
Types are contained in module DownloaderTypes
InputFile
url: string
output_path: string | null //This attribute overwrite output_directory defined on options only for this file.
OutputFile
url: string
path: string | null
retry_times: number
response: Response | null
Options
retry_times: number = 3
debug_mode: DebugMode = DebugMode.LOG
output_directory: string = './download/'
Response
status: Status;
message: any;
enum Status{
OK,
KO
}
enum DebugMode{
NONE,
DEBUG,
LOG
}