NodeJS library to communicate with an objectstore

Table of Contents

• Features
• ETag Support
• Installing
• Configuration
• Implementation
• License

Features

• Fully integrated in CAP as a Service
• Support S3 objectstore and local objectstore for development testings
• Autodeletion if corresponding entity entry is been deleted (Entity which uses File)
• Autohandling streaming, if content is requested
• Testfiles possible
• Lazy deletion of objects and only executed, if transaction is successfull
• Auto configuration with cds plugin feature
• Automatic ETag from S3 for file integrity validation

ETag Support

Files entity includes an eTag field for file integrity validation.

Features:

• eTag field (MD5 hash from S3, nullable for legacy files)
• S3Client and LocalClient automatically return ETag after upload
• ETag is stored in database for integrity validation

Usage:

// API unchanged - works exactly as before
await objectstore.create(content, { filename: 'test.txt' })

// Now with automatic ETag storage
const file = await objectstore.read(ID)
console.log(file.eTag)  // MD5 hash from S3

Notes:

• Existing files will have NULL eTag (handled gracefully)
• New uploads automatically get MD5 hash from S3 (single-part uploads)
• No migration required

Installing

Using npm:

$ npm install @amag-ch/cds-objectstore

Using yarn:

$ yarn add @amag-ch/cds-objectstore

Configuration

{
    "cds": {
        "requires": {
            "objectstore": {
                "impl": "@amag-ch/cds-objectstore",
                "kind": "objectstore",
                "S3_config": { //S3 specific configuration options, e.g.
                    "maxAttempts": 2,
                    "requestHandler": {
                        "connectionTimeout": 30000,
                        "requestTimeout": 30000,
                        "throwOnRequestTimeout": true
                    }
                }
            }
        }
    }
}

The name and kind must be named objectstore, otherwise CAP cannot inject the credentials from environment

Custom metadata is not persisted in the database by default. Enable persistence and metadata queries with persist_metadata:

{
    "cds": {
        "requires": {
            "objectstore": {
                "persist_metadata": true
            }
        }
    }
}

The metadata database table is only deployed when this package option is enabled. For production object-store bindings, startup also schedules a background job that fetches and persists metadata for existing files that have no metadata rows yet.

Variant with local objectstore for development testings

{
    "cds": {
        "requires": {
            "objectstore": {
                "impl": "@amag-ch/cds-objectstore",
                "kind": "local-objectstore",
                "[production]": {
                    "kind": "objectstore",
                }
            }
        }
    }
}

The local objectstore is saved in folder ~/.cds-objectstore

Standard configuration (cds-plugin)

{
    "cds": {
        "requires": {
            "objectstore": {
                "impl": "@amag-ch/cds-objectstore",
                "kind": "local-objectstore",
                "[production]": {
                    "kind": "objectstore"
                }
            }
        }
    }
}

Implementation

Database Schema

using {amag.common.objectstore.File as File} from '@amag-ch/cds-objectstore';

entity MyEntity {
    key ID      : UUID;
        file    : File
}

File is definied as Composition. Means if entry of MyEntity is been deleted, also the connected File would be deleted. File is per default defined as attachment (@Core.ContentDisposition.Type: 'attachment'), but could be overwriten with own annotations

Service for add or read files

const objectstore = await cds.connect.to('objectstore')

// Create file - ETag automatically obtained from S3
const ID = await objectstore.create('Any File Content', {
    filename: 'Testfile.txt',
    contentType: 'text/plain',
    metadata: {
        source: 'example',
        category: 'document'
    }
})

const file = await objectstore.read(ID)
console.log(file.modifiedAt)
console.log(file.eTag)  // MD5 hash from S3

// Query object-store metadata (metadata keys are returned lowercase)
const { metadata } = await objectstore.getMetadata(ID)
console.log(metadata.source)

// Requires persist_metadata: true. All supplied entries must match.
const IDs = await objectstore.findByMetadata({
    source: 'example'
})

const stream = await objectstore.readContent(ID)

Testing

Create a folder (e.g. test/objectstore) with all needed files named by ID, which is also used in test/data/amag.common.objectstore-Files.csv.

With cds-plugin mode this folder is automatically registered.

Otherwise add file test/init.js with following code snippet to register your folder.

module.exports = async () => {
    require('@amag-ch/cds-objectstore').testing.register(`${__dirname}/objectstore`)
}

License

MIT