npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

high-availability-object-storage

v0.4.0

Published

High available, performant, and tiny Node SDK Client for OpenStack Swift Object Storage

Downloads

12

Readme

High available Node Client for OpenStack Switf Object Storage

GitHub release (latest by date) Documentation

High availability, Performances, and Simplicity are the main focus of this tiny Node SDK to request the OpenStack Object Storage API. It was initially made to request the OVHCloud Object storage, but it can be used for any OpenStack Object Storage.

Highlights

  • 🦄 Simple to use - Only 5 methods: uploadFile, deleteFile, listFiles, downloadFile and request for custom requests.
  • 🌎 High availability - Initiate the SDK with a list of object storages credentials, and the SDK will switch storage if something goes wrong (Server/DNS not responding, timeout, error 500, too many redirection, authentication error, and more...).
  • Reconnect automatically - If a request fails due to an authentication token expiration, the SDK fetches a new authentication token and retry the initial request with it.
  • 🚀 Performances - Less than 500 lines of code with only 2 dependencies simple-get and debug.
  • 100% tested - Battle-tested against hundreds of GBs of file uploads & downloads

Install

1. Prior installing

you need a minimum of one object storage container, or you can synchronize Object Storages containers in order to access same objects if a fallback occur:

  • Sync 2 containers: 1 <=> 2. They would both need to share the same secret synchronization key.
  • You can also set up a chain of synced containers if you want more than two. You would point 1 -> 2, then 2 -> 3, and finally 3 -> 1 for three containers. They would all need to share the same secret synchronization key. Learn more on the OpenStack documentation or on the OVHCloud documentation.
  1. Install the swift-pythonclient, an easy way to access Storages is with the Swift command line client, run on your terminal:
$ pip install python-swiftclient
  1. Download the OpenStack RC file on the OVH account to change environment variables. Tab Public Cloud > Users & Roles > Pick the user and “Download OpenStack’s RC file”
  2. Open a terminal, load the contents of the file into the current environment:
$ source openrc.sh
  1. In order for the containers to identify themselves, a key must be created and then configured on each container:
$ sharedKey=$(openssl rand -base64 32)
  1. See which region you are connected to:
env | grep OS_REGION
  1. Retrieve the Account ID AUTH_xxxxxxx of the destination container in order to configure the source container:
destContainer=$(swift --debug stat containerBHS 2>&1 | grep 'curl -i.*storage' | awk '{ print $4 }') && echo $destContainer
  1. Change to the source region:
OS_REGION_NAME=RegionSource
  1. Upload the key and the destination sync url to the source container:
$ swift post -t ‘//OVH_PUBLIC_CLOUD/RegionDestination/AUTH_xxxxxxxxx/containerNameDestination’ -k "$sharedKey" containerNameSource
  1. You can check that this has been configured by using the following command:
$ swift stat containerName
  1. You can check if the synchronization worked by listing the files in each of the containers:
$ OS_REGION_NAME=RegionSource && swift list containerName
$ OS_REGION_NAME=RegionDestination && swift list containerName

2. Install the package with your package manager:

$ npm install --save high-availability-object-storage
// od
$ yarn add high-availability-object-storage

API Usage

Connection

Initialise the SDK with one or multiple storage, if something goes wrong, the next region will take over automatically. If any storage is available, an error message is returned Error: Object Storages are not available.

const storageSDK = require('high-availability-object-storage');

let storage = storageSDK([{
  authUrl    : 'https://auth.cloud.ovh.net/v3',
  username   : 'username-1',
  password   : 'password-1',
  tenantName : 'tenantName-1',
  region     : 'region-1'
},
{
  authUrl    : 'https://auth.cloud.ovh.net/v3',
  username   : 'username-2',
  password   : 'password-2',
  tenantName : 'tenantName-2',
  region     : 'region-2'
}]);

storage.connection((err) => {
  if (err) {
    // Invalid credentials
  }
  // Success, connected!
})

Upload a file

const path = require(path);

/** SOLUTION 1: The file content can be passed by giving the file absolute path **/
storage.uploadFile('container', 'filename.jpg', path.join(__dirname, './assets/file.txt'), (err) => {
  if (err) {
    // handle error
  }
  // success
});

/** SOLUTION 2: A buffer can be passed for the file content **/
storage.uploadFile('container', 'filename.jpg', Buffer.from("File content"), (err) => {
  if (err) {
    // handle error
  }
  // success
});

/** SOLUTION 3: the function accepts a optionnal fourth argument `option` including query parameters and headers. List of query parameters and headers: https://docs.openstack.org/api-ref/object-store/?expanded=create-or-replace-object-detail#create-or-replace-object **/
storage.uploadFile('container', 'filename.jpg', Buffer.from("File content"), { queries: { temp_url_expires: '1440619048' }, headers: { 'X-Object-Meta-LocationOrigin': 'Paris/France' }}, (err) => {
  if (err) {
    // handle error
  }
  // success
});

Download a file

storage.downloadFile('templates', 'filename.jpg', (err, body, headers) => {
  if (err) {
    // handle error
  }
  // success, the `body` argument is the content of the file as a Buffer
});

Delete a file

storage.deleteFile('templates', 'filename.jpg', (err) => {
  if (err) {
    // handle error
  }
  // success
});

List objects from a container

/**
 * SOLUTION 1
 **/
storage.listFiles('templates', function (err, body) {
  if (err) {
    // handle error
  }
  // success
});

/**
 * SOLUTION 2
 * Possible to pass queries and overwrite request headers, list of options: https://docs.openstack.org/api-ref/object-store/? expanded=show-container-details-and-list-objects-detail#show-container-details-and-list-objects
 **/
storage.listFiles('templates', { queries: { prefix: 'prefixName' }, headers: { Accept: 'application/xml' } }, function (err, body) {
  if (err) {
    // handle error
  }
  // success
});

Get file metadata

Shows object metadata. Checkout the list of headers.

storage.getFileMetadata('templates', 'filename.jpg', (err, headers) => {
  if (err) {
    // handle error
  }
  /**
   * Returned headers: {
   *  Content-Length: 14
   *  Accept-Ranges: bytes
   *  Last-Modified: Thu, 16 Jan 2014 21:12:31 GMT
   *  Etag: 451e372e48e0f6b1114fa0724aa79fa1
   *  X-Timestamp: 1389906751.73463
   *  X-Object-Meta-Book: GoodbyeColumbus
   *  Content-Type: application/octet-stream
   *  X-Trans-Id: tx37ea34dcd1ed48ca9bc7d-0052d84b6f
   *  X-Openstack-Request-Id: tx37ea34dcd1ed48ca9bc7d-0052d84b6f
   *  Date: Thu, 16 Jan 2014 21:13:19 GMT
   *  X-Object-Meta-Custom-Metadata-1: Value
   *  X-Object-Meta-Custom-Metadata-2: Value
   * }
   * // Details: https://docs.openstack.org/api-ref/object-store/?expanded=show-object-metadata-detail#show-object-metadata
   */
});

Set file metadata

To create or update custom metadata, use the "X-Object-Meta-name" header, where name is the name of the metadata item. The function overwrite all custom metadata applied on the file. Checkout the list of headers availables.

storage.setFileMetadata('templates', 'filename.jpg', { headers: { 'Content-Type': 'image/jpeg', 'X-Object-Meta-LocationOrigin': 'Paris/France', 'X-Delete-At': 1440619048 }} (err, headers) => {
  if (err) {
    // handle error
  }
  // success
});

Custom request

The request function can be used to request the object storage with custom options. Prototype to get the data as Buffer:

request(method, path, { headers, queries, body }, (err, body, headers) => {}).

Prototype to get the data as Stream, set the option stream:true:

request(method, path, { headers, queries, body, stream: true }, (err, dataStream) => {})`.

The base URL requests by default the account, passing an empty string will request the account details. For container requests, pass the container name, such as: /{container}. For file requests, pass the container and the file, such as: /{container}/{filename}. Object Storage Swift API specification: https://docs.openstack.org/api-ref/object-store/

The request function automatically reconnects to the Object Storage or switch storage if something goes wrong.

Example of custom request, bulk delete file from a customerDocuments container:

  const _headers = {
    'Content-Type': 'text/plain',
    'Accept'      : 'application/json'
  }

 storage.request('POST', '/customerDocuments?bulk-delete', { headers: _headers, body: 'file1\nfile2\n' }, (err, body, headers) => {
  /**
  * body: {
  *  "Number Not Found": 0,
  *  "Response Status": "200 OK",
  *  "Errors": [],
  *  "Number Deleted": 2,
  *  "Response Body": ""
  * }
  */
  done();
});

Log

The package uses debug to print logs into the terminal. To activate logs, you must pass the DEBUG=* environment variable. You can use the setLogFunction to override the default log function. Create a function with two arguments: message as a string, level as a string and the value can be: info/warning/error. Example to use:

storage.setLogFunction((message, level) => {
  console.log(`${level} : ${message}`);
})

Run tests

Install

$ npm install

To run all the tests:

$ npm run test

🤝 Contributing

Contributions, issues and feature requests are welcome!

Feel free to check issues page.

Show your support

Give a ⭐️ if this project helped you!

👤 Author