Imago Azure Storage

This is an opinionated wrapper for Microsoft's azure-storage package for working with Azure storage. It provides a more convenient syntax and async support.

Performance

It is best to cache the files stored in the cloud storage, because the retrieval times are 20-200 ms from within the same region, and 200-600 ms if the data is transferred between regions.

Initialization

The constructor takes the storage account name (e.g. myaccount) and the storage service key that you can find in Azure Portal after creating a storage account.

These credentials are better stored in the .env file. Then you can initialize the service as follows:

    const { Blobs } = require('imago-azure-storage');
    global.blobs = new Blobs(
      process.env.AZURE_STORAGE_ACCOUNT, process.env.AZURE_STORAGE_ACCESS_KEY);

The reason to store the blobs service in the global variable is to avoid reestablishing the connection to the server each time the service is accessed from different parts of the code.

Methods

async initializeContainers(containers)

Initializes the containers, meaning it checks that the containers are created, or creates them.

The containers is an object in the following format:

const AZURE_BLOB_STORAGE_CONTAINERS = {
  MY_CONTAINER_1: { name: 'my-container-1', type: 'private' },
  MY_OTHER_CONTAINER: { name: 'my-other-container', type: 'public' },
};

The key is irrelevant and is for the programmer's convenience when calling other methods like writeFile. The name is the container name in Azure. The type can be private or public (allows anyone to download files via a public URL).

async writeFile(container, filename, content)

Write a file to storage.

async append(container, filename, content)

Append to a file in storage.

async readFile(container, filename)

Read a file from storage.

async exists(container, filename)

Check if file exists in storage.

Since each operation costs money, in some cases you can simply try to read a file using readFile(), and catch an exception if it doesn't exist.

getPublicDownloadLink(container, filename)

Get a public download URL for a filename. The container type must be set to public when it was first created.

Azure Tables

Azure Table storage is also supported.

Methods

constructor(storageAccount, storageAccessKey)

async initializeTables(tables)

Create tables if they don't exist. tables is an array of strings (table names).

async retrieveEntity(table, partition, key)

Read an entity from table.

async retrieveAllEntitiesByPartitionKey(table, partition, maxPages = 0)

Return all entities from a table that have the specified partition key.

The maxPages controls the maximum number of queries sent to the server to retrieve data (each query may up to 1000 rows). Set to 0 to retrieve all data. Note that maxPages = 1 does not guarantee that some data will be returned: some queries may return empty data because the requested data resides on another server. For safety, retrieve a few pages and then trim the data if necessary.

async retrieveEntireTable(table, maxPages = 0)

Returns all data from a table. Each 1000 entities will be a separate API request (charged by Azure), and retrieving a large number of entities may take a significant time. Do it only when the table is small enough, at most a few thousand records.

The maxPages controls the maximum number of queries sent to the server to retrieve data (each query may up to 1000 rows). Set to 0 to retrieve all data. Note that maxPages = 1 does not guarantee that some data will be returned: some queries may return empty data because the requested data resides on another server. For safety, retrieve a few pages and then trim the data if necessary.

async saveStringOnlyEntity(table, partition, key, data)

Save or replace entity with the specified key in database. data is an object where each key is a column name.

async saveStringOnlyEntity(table, data)

Same as above. The data must contain the PartitionKey and RowKey properties.

async query(table, partition, conditions, maxPages = 0)

Queries a table based on the column values. conditions is an array of arrays, where each condition is an array containing two or more elements, for example ['userName eq ?', 'Anton'] or ['hitCount >= ? and hitCount <= ?', '2', '100'].

If partition is null or empty string, then all partitions will be searched.

The maxPages controls the maximum number of queries sent to the server to retrieve data (each query may up to 1000 rows). Set to 0 to retrieve all data. Note that maxPages = 1 does not guarantee that some data will be returned: some queries may return empty data because the requested data resides on another server. For safety, retrieve a few pages and then trim the data if necessary.

async deleteEntity(table, partition, key)

Deletes the specified entity from table by row key.

Azure Queues

Example

const { Queues } = require('imago-azure-storage');
const q = new Queues(storageAccount, storageKey);

const QUEUE_NAME = 'testqueue';

(async() => {
    await q.initializeQueues([QUEUE_NAME]);
    
    // Write:
    await q.put(QUEUE_NAME, { test: 12345 });
    
    // Retrieve:
    let items = await q.fetch(QUEUE_NAME);
    for (let { item, message } of items) {
        console.log(item); // { test: 12345 }

        // Delete from queue:
        await q.commit(message);
    }
})();

async initializeQueues(queues)

Creates queues if they don't exist yet. queues is an array of strings which are names of the queues.

async put(queue, message)

Places a message into the queue. message can be string or object, but must be less than 48 KB in size (for objects - when serialized).

async fetch(queue, numOfMessages, visibilityTimeout)

Locks and fetches available items from queue.

Optional arguments:

• numOfMessages: How many items to retrieve (up to 32)
• visibilityTimeout: Timeout in seconds after which messages will be requeued if not committed. Default 30 seconds.

It returns the array of objects, where each object contains item which is your data that you placed into the queue, and message that can be used to delete the item from the queue once your processing is done. If you don't delete the item, it will be placed back into the queue when the timeout expires.

commit(message)

Deletes an item from the queue. message must be the object returned as part of the data from fetch().