npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@adobe/httptransfer

v3.4.1

Published

Efficient File Transfer Module for NodeJS

Downloads

4,421

Readme

node-httptransfer

Version

Introduction

The node-httptransfer package is designed to easily and correctly transfer file content from HTTP(S) urls to HTTP(S) urls and between HTTP(S) urls.

The lower-level stream API allows you to transfer content from a URL to any writable stream, and similarly transfer any readable stream to a URL.

The higher-level file API abstracts away the streams and allows you to transfer content to/from files on disk.

The node-httptransfer package requires the async/await features and is built using the node-fetch-npm package.

To use block transfer capabilities, the upload/download servers must support the range header.

Installation

npm i @adobe/httptransfer

Using streams

Download a stream:

const { downloadStream } = require('@adobe/httptransfer');
async main() {
    const stream = fs.createWriteStream('test.png');
    await downloadStream('http://my.server.com/test.png', stream);
}

Upload a stream using PUT:

const { uploadStream } = require('@adobe/httptransfer');
async main() {
    const stream = fs.createReadStream('test.png');
    await uploadStream(stream, 'http://my.server.com/test.png');
}

Using files

Download a file:

const { downloadFile } = require('@adobe/httptransfer');
async main() {
    await downloadFile('http://my.server.com/test.png', 'test.png');
}

Upload a file using PUT:

const { uploadFile } = require('@adobe/httptransfer');
async main() {
    await uploadFile('test.png', 'http://my.server.com/test.png');
}

Upload a file to multiple URLs using PUT (used by AEM multi-part upload):

const { uploadAEMMultipartFile } = require('@adobe/httptransfer');
async main() {
    await uploadAEMMultipartFile('test.png', {
        urls: [ "http://my.server.com/test.png.1", "http://my.server.com/test.png.2" ],
        maxPartSize: 1000000
    });
}

Using block upload/downland to upload/download files concurrently

const { downloadFileConcurrently } = require('@adobe/httptransfer');
async main() {
    await downloadFile('http://my.server.com/test.png', 'test.png');
}

Upload a file using PUT:

const { uploadFileConcurrently } = require('@adobe/httptransfer');
async main() {
    await uploadFile('test.png', 'http://my.server.com/test.png');
}

Upload a file to multiple URLs using PUT (used by AEM multi-part upload):

const { uploadMultiPartFileConcurrently } = require('@adobe/httptransfer');
async main() {
    await uploadMultiPartFileConcurrently('test.png', {
        urls: [ "http://my.server.com/test.png.1", "http://my.server.com/test.png.2" ],
        maxPartSize: 1000000
    });
}

Upload multiple files to multiple URLs using PUT:

const { uploadFilesConcurrently } = require('@adobe/httptransfer');
async main() {
    await uploadFilesConcurrently([{
        filepath: 'file1.png',
        target: {
            urls: [ "http://my.server.com/file1.png.1", "http://my.server.com/file1.png.2" ],
            maxPartSize: 1000000
        }
    }], {
        filepath: 'file2.png',
        target: {
            urls: [ "http://my.server.com/file2.png.1", "http://my.server.com/file2.png.2" ],
            maxPartSize: 1000000
        }
    }]);
}

Assuming test.png is 1,800,000 bytes this will upload the first 1,000,000 bytes to http://my.server.com/test.png.1 and the next 800,000 bytes to http://my.server.com/test.png.2.

Testbed

A CLI tool testbed is provided to try out the node-httptransfer functionality. It supports uploading, downloading, and transferring file content. It also supports Azure Blob stores through Shared Access Signature (SAS) urls created on the fly.

The tool is not intended to be useful on its own, only to test out new features or debug issues.

Build

cd testbed/
npm install

Azure credentials

export AZURE_STORAGE_ACCOUNT=<storage account name from https://portal.azure.com>
export AZURE_STORAGE_KEY=<storage key from https://portal.azure.com>

Examples

Download an image from a website:

node index.js https://website.com/path/to/image.gif image.gif

Download blob.txt from azure:

node index.js azure://container/path/to/blob.txt blob.txt

Upload blob.txt to azure:

node index.js blob.txt azure://container/path/to/blob.txt

Upload blob.txt in 10,000 byte blocks:

node index.js --max 10000 blob.txt azure://container/path/to/blob.txt

Copy blob.txt within a container:

node index.js azure://container/path/to/blob.txt azure://container/path/to/target.txt

End-to-End Tests

The module provides some integration-style tests for verifying basic transfer functionality with an Adobe Experience Manager instance. To run the tests:

  • Create a .env file by following the instructions in .env_example.
  • Run the tests by executing npm run e2e from the root directory of the repository.

End-to-End block upload/download Tests

If you want to just run the block upload/download tests, you only need Azure credentials:

export AZURE_STORAGE_ACCOUNT=<storage account name from https://portal.azure.com>
export AZURE_STORAGE_KEY=<storage key from https://portal.azure.com>
export AZURE_STORAGE_CONTAINER_NAME=<storage container name>

The run npm run e2e-block

Contributing

Contributions are welcomed! Read the Contributing Guide for more information.

Licensing

This project is licensed under the Apache V2 License. See LICENSE for more information.