Nest Storage Manager

The Nest Storage Manager manages multiple storage engines and exposes the StorageService as the primary interface for file operations.

Installation

To use Nest Storage in your Nest.js application, install the package via npm:

npm install @biocomputingup/nest-storage-manager

Usage

1. Import StorageManagerModule

import { StorageManagerModule, Engine } from '@biocomputingup/nest-storage-manager'
import { LocalStorageEngine, S3StorageEngine } from '@biocomputingup/nest-storage-manager/engines';
const [PublicStorage, PrivateStorage] = ['public', 'private'];

@Module({
  imports :[
    // Other imports
    StorageManagerModule.forRoot({
      default: PublicStorage,
      engines: [
        {
          name: PrivateStorage,
          engine: YourImplementationOfStorageEngine // Example for custom engines
        },
        {
          name: PublicStorage, // You can use the factory pattern here as well
          imports: [ConfigModule],
          inject: [ConfigService],
          useFactory: (config: ConfigService): StorageEngine => new LocalFsEngine({
            folderPath: 'path/to/storage/folder',
            baseUrl: 'http://localhost:3000',
          })
        },
      ],
    });
  ]
})
class AppModule {}

2. Inject and use StorageService or each engine individually

From anywhere in your application, inject and use the StorageService to perform operations:

import { Injectable } from '@nestjs/common';
import { StorageService } from '@biocomputingup/nest-storage-manager';

@Injectable()
export class YourService {
  constructor(private readonly storageService: StorageService) {}

  // Use storageService methods for file and folder operations
  // Example: this.storageService.put('path/to/file.txt', bufferData);
}

Or inject each storage engine individually

import { Injectable } from '@nestjs/common';
import { StorageService } from '@biocomputingup/nest-storage-manager';

@Injectable()
export class YourService {
  constructor(@InjectEngine(PrivateStorage) private readonly engine: Engine<YourEngineClass>) {}

  // Use the engine
  // Example: this.storageService.put('path/to/file.txt', bufferData);
}

Included Storage Engines

LocalStorageEngine

Async Factory Provider implementation

{
	name: 'private',
	inject: [ConfigService],
	useFactory: (config: ConfigService) => ({
      engine: new LocalFsEngine({
        folderPath: config.get('/path/to/folder'),
        baseUrl: 'http://localhost:3000', // The URL prefix, if you plan to serve these files the storage engine will prefix them with this path when returning public url for a file e.g engine.getPublicUrl('filename')
      }),
  }),
},

Regular Value Provider implementation

{
  name: 'privateStorage',
  engine: new LocalFsEngine({
    // Config here
  })
}

S3StorageEngine

// in the app module the config is the following

{
	name: 'myS3Storage',
	engine: new S3Engine({
    bucket: 'bucket-name',
    proxy: { // Optional in case the S3 instance is behind a proxy
      proxyEndpoint: 'http://my-url' // The public facing URL endpoint associated with S3 (used to build public URLs)
      bucketEndpoint: 'http://S3-url' // The endpoint of the files returned by S3
    }
  }, new S3Client() /* An instance of S3Client from @aws-sdk/client-s3 */ ),
}

Custom Storage Engine

If your use case doesn't fit the already provided storage engines. You can implement your own and use it as any other engine

// All you need to do is to pass a class that implements this interface

/**
 * Interface representing a storage engine.
 *
 * @template EngineConfig - The type of the configuration object for the storage engine.
 */
export interface StorageEngine {
  /**
   * Retrieves the data at the specified path.
   *
   * @param path - The path to the data.
   * @returns A promise that resolves with the data as a Buffer.
   */
  get(path: string): Promise<Buffer>;

  /**
   * Retrieves a stream for the data at the specified path.
   *
   * @param path - The path to the data.
   * @returns A stream of the data, or a promise that resolves with the stream.
   */
  getStream(path: string): Stream | Promise<Stream>;

  /**
   * Retrieves a public URL for the data at the specified path.
   *
   * @param path - The path to the data.
   * @returns A promise that resolves with the public URL as a string.
   */
  getPublicUrl(path: string, ...otherArgs: any[]): Promise<string>;

  /**
   * Stores the given data at the specified path.
   *
   * @param path - The path where the data should be stored.
   * @param data - The data to store, as a Buffer or a ReadableStream.
   * @returns A promise that resolves when the data has been stored.
   */
  put(path: string, data: Buffer | NodeJS.ReadableStream): Promise<void>;

  /**
   * Deletes the data at the specified path.
   *
   * @param path - The path to the data to delete.
   * @returns A promise that resolves when the data has been deleted.
   */
  delete(path: string): Promise<void>;

  /**
   * Lists the paths of all data items with the specified prefix.
   *
   * @param prefix - The prefix to filter the paths.
   * @returns An async generator that yields the paths as strings.
   */
  listPaths(prefix: string): AsyncGenerator<string, void, unknown>;
}

constructor(private storageService: StorageService) {}

async uploadFile() {
	const path = await this.storageService.put('public', 'some/path', myData);
}

Use getStream, delete, and other methods in the same way, passing either an engine name and path or a canonical URL.

Example Imports

import { LocalStorageModule, StorageService } from '@biocomputingup/nest-storage-manager';

Canonical URLs

Canonical URLs map for storage locations and paths, making it easier to share or locate resources independently of the chosen engine. They typically follow the pattern: storage://{engineName}/path/to/file.

CanonicalURLs are returned when performing a put delete and can be used in the get operation of the storageManager to be used as file references in your DB

StorageService API

• put(engine, path, data): Uploads data.
• getStream(engine, path): Retrieves a stream for reading.
• delete(engine, path): Removes a file.
• list(engine, path?): Lists files within a path if provided.
• exists(engine, path): Checks if a file exists.

Adjust the engine name and path or use a canonical URL for all operations.

Setup Guide

1. Install and import the module in your main application module.
2. Configure your storage engines as needed.
3. Inject the StorageService wherever file operations are required.

Usage Examples

put

async saveData(blob: Blob) {
	await this.storageService.put('public', 'docs/report.pdf', blob);
}

getStream

async readFile() {
	const stream = await this.storageService.getStream('private', 'user/data.json');
	// Process the stream as needed
}

delete

async removeFile() {
	await this.storageService.delete('public', 'old/logs.txt');
}