Document Service

Document Service allows consumers to get and update data about Endpoint Documents.

Table of contents

• Description
• Installation / setup
  • Development
  • Run Scripts
  • Document Service SLA
  • Swagger Links
  • Production
  • Environment variables
• Testing

Description

The Document Service manages the state related to documents that are shared between our customers and operations teams. All requests to create, access, or modify documents - by both internal and external systems - are serviced here.

Further documentation on the service can be found here.

Note: In order to support multi-tenancy in our platform, apps/document-service/src/AppModule.ts#32 configures the application to utilize the common endpoint auth guard. It is configured to only be used in non-local development environments to enable better developer experience.

Technologies

This service is written in TypeScript, using the NestJS framework. It is packaged and run as a Docker container, which internally contains the NodeJS runtime.

Other significant technologies to note are:

• ExpressJS the web framework used (read more about Nest and Express here)
• DynamoDB the database service used
• PostgreSQL the relational database service used
• TypeORM manages the repository layer and migrations.

Installation

In order to run the service locally, you will at minimum need to have docker installed. See local development instructions for details. If you want to run scripts from the package.json outside of the docker container, then you will need to ensure that your local node version is compatible with the version range listed in package.json -> engines -> node.

Database (PostgreSQL)

The service is backed by PostgreSQL. TypeORM manages the repository layer and migrations.

Run the database locally.

docker-compose up -d db

Migration run scripts

The run scripts in package.json allow easy execution of the 5 basic migration run scripts: create, generate, show, run and revert,

Show pending migrations

In the local environment

yarn orm:mig:show:local

It shows all ready-for-execution migrations. It iterates all created files in the dist/migrations folder.

Notice: you must build the app to get the javascript-format migrations.

Add a new migration

Empty migration:

yarn orm:mig:create -n <migration_name>

It generates a migration class with empty methods under the following path: ./migrations/<numeric_timestamp>-<migration_name>.ts, with class name <migration_name><numeric_timestamp>

Example migrations/1655932765632-MigrationName.ts

Revert migrations

In the local environment

yarn orm:mig:revert:local

It reverts all the already-executed queries in a previous run. e.g. the ones defined in the down() method of the migration classes.

In other environments

Ask devops to perform the following actions:

1. backup the database
2. navigate to the running container via shell
3. DEPLOY_ENV= yarn orm:mig:revert

Local Development

It is possible to execute the Document Service almost entirely using local infrastructure. This is made possible by auto-mocking several external services and using localstack to mock AWS infrastructure.

If one has not been created already, create a .env file at the root of the application. This should be automatically git-ignored. The file should contain values for the following variables:

1. (Required) Set NPM_TOKEN=<your NPM token>
2. (Optional) Set the LAUNCH_DARKLY_SDK_KEY for local development in the root level .env file. This can be found in AWS secrets manager.

To start the document-service and all necessary infrastructure, run

docker-compose up

This will start the service using local as the DEPLOY_ENV and watching for any changes via the start:dev script.

To start the document-service in debug mode, run

1. start the document-service and all necessary infrastructure

docker-compose up

2. list all running containers

docker container ls

3. scan the list for the container ID of document-service
4. stop only document-service container

docker container stop <container-id-of-document-service>

5. start the document-service in debug mode

   DEPLOY_ENV={ENV} yarn start:dev

Known Issues

• The Token Exchange Service is not mocked and is still being used by the auth guard. We haven't found a way yet to mock a valid JWT response from the service, and as a result the service's call to OCR Service is failing. This call initiates a side-effect and should not inhibit testing of the Document Service.

Run Scripts

Run dev

DEPLOY_ENV=local yarn start:dev

Scaffolds commit messages and enforces required format using Endpoint's Commitizen configuration.

yarn commit

Runs linter

yarn lint

Fixes linter errors

yarn lint:fix

Resets node_modules, clears yarn cache, and reinstalls node_modules

yarn reset-modules

Runs integration tests using cypress.

yarn run test:cypress

Document Service SLA

SLA

Swagger Links

• Local
• Development
• Integration
• Staging

Production

Use this space to describe any requirements that need to be set up before the app can be deployed to production.

Environment variables

DEPLOY_ENV

This variable designates which of our environments this service is running in (i.e. local, development, integration, test, sandbox, staging, production).

DYNAMO_REGION

The AWS region to connect to for dynamo. In development or testing environments with a localstack dynamo instance this value should be local. TODO: Document why this env variable needs to be set and what precisely it influences.

PG_HOST

The URL of the PostgreSQL (RDS) host to connect to.

• For environments migrated into environment-specific AWS accounts, this value is specified in the corresponding file in the ep-k8s repo
  • Example in Development
• For environments still running in the "legacy" AWS account, this vlaue is specified in the corresponding document-service/config/.ENV_NAME.env file

PG_PASSWORD

The password for the specified PG_HOST. This value is configured and stored in AWS secrets manager.

SENTRY_DSN

The sentryDSN url from Sentry to enable sentry logging of errors, if not set it will disable sentry reporting for errors.

S3_SERVER_SIDE_ENCRYPTION

The type of encryption to set for objects stored in S3. Information about server side encryption (SSE) can be found in the S3 SSE documentation.

S3_REGION

The AWS region where the S3 bucket for this service resides.

S3_BUCKET_NAME

The name of the S3 bucket to connect to.

ENABLE_REQUEST_LOGGING

Enable logging for incoming requests through the Endpoint Core Module when set to true (see docs).

PROVIDER_TEMPLATES_SYNCHRONIZE_CIRCUIT_BREAKER_MILLISECONDS

Max number of milliseconds of timeout for synchronizing document templates from provider

Testing

Unit Tests

To run unit tests:

yarn test

Testing DynamoDB Queries

We use jest-dynalite to enable validation of dynamo queries.

Because not all unit test suites need an instance of dynalite, it is only spun up by manually including it in your tests. The simplest way is to include the following import at the top of your test suite:

import "jest-dynalite/withDb";

This will add the required setup and teardown hooks for dynalite. If you need more granular control, see further options in the Advanced Setup instructions.

Cypress Tests

User Admin credentials are required to run the cypress test. The credentials are stored in the Cypress Test User (dev, int, stg) entry in the Engineering vault.

1. Create a cypress.env.json file at the project root. This is already git-ignored and thus safe for secrets.
2. Add the following, replace {username} and {password} with the values retrieved from the Engineering Vault

{
  "ADMIN_USER_NAME": "{username}",
  "ADMIN_USER_PASSWORD": "{password}"
}

Example: run the following in two shells to run the cypress tests locally against a local service process using the development config. Cypress is setup to run integration test.

yarn run test:cypress

By default, the test will run against your local server. An env variable can be passed to cypress to specify which environment you want to run the test (development, integration, staging).

yarn run test:cypress -e name=staging

Note:

• For non-production environments a transactionId and documentId will be auto provided as env vars when running the cypress run script. For more details see file /cypress/plugins/setUpTransaction/setUpTransaction.ts.
• If you need to add a new dependency to the E2E test cases, you will need to update test-automation.yml pipeline to install the needed dependency for the test runner

Adding and running tagged test cases

Tags help us to organize and run tests based on the type of case they are. Right now, we only tag integration test cases. We use the tagging from the cypress/grep package to help us tag and run cases.

Adding tags to a test case

To add a tag to your test case you'll need to add { tags: ['@integration'] } to your it statement. Ex.

it('C290718 - Open Order - Prelim conversion to Escrow', { tags: ['@integration'] }, () => {

Running tagged test cases

To run tagged test cases you'll need to add grepTags={tags} to your cypress run script, where {tags} is replaced with the text of your tag

Ex.

yarn cypress open --env name=integration,grepTags='@integration'

Performance Tests

We use K6 for our performance tests. You'll need to have K6 installed on your machine before attempting to execute the tests.

Scripts

Upload Latency

This script measures the time between a file upload completing and that file being available in Verse. It accomplishes this by uploading a file to document-service and then polling the service for that document's etag. It is configured to simulate 5 virtual users and have each user upload 100 files for a total of 500 uploads.

Environment Variables

The following environment variables are required:

• TX - a transaction ID
• TOKEN - a token to include in the Authorization header of outgoing requests
• FILE - a path to a file on your local machine to upload

Running the Test

Make sure you're connected to the VPN, then run:

k6 run -e TX="TRANSACTION_ID" -e TOKEN="TOKEN" -e FILE="PATH_TO_FILE" upload-latency.js

Be sure to substitute a valid transaction ID and authorization token for the TX and TOKEN environment variables.

Download Availability

This script measures the availability of downloading a file. It is configured to simulate 5 virtual users and have each user download 300 files for a total of 1500 downloads. This script can set up a document, upload a document and then download it.

Environment Variables

This script uses the config.json file to set its environment variables:

• EP2_URL - ep2-services endpoint (auth token generator)
• DOCUMENT_SERVICE_URL - document-service url
• FILE_PATH - file to upload and download
• COGNITO_CLIENT_ID - cognito client id to get Auth Tokens
• COGNITO_CLIENT_SECRET - cognito client secret to get Auth Tokens

Running the Test

Make sure you're connected to the VPN, then run:

yarn build

k6 run download-availability.js

Upload Availability

This script measures the availability of uploading a file. It is configured to simulate 5 virtual users and have each user download 20 files for a total of 100 uploads. This script can set a document and upload the document's content

Environment Variables

This script uses the config.json file to set its environment variables:

• EP2_URL - ep2-services endpoint (auth token generator)
• DOCUMENT_SERVICE_URL - document-service url
• FILE_PATH - file to upload and download
• COGNITO_CLIENT_ID - cognito client id to get Auth Tokens
• COGNITO_CLIENT_SECRET - cognito client secret to get Auth Tokens

Running the Test

Make sure you're connected to the VPN, then run:

yarn build

We recommend execute just a single scenario to avoid uploading a lot of files to our environments. Scenario list:

• single_user (a single user uploading 100 files)
• single_user_parallel (a single user uploading 100 files in parallel in 30 seconds)
• five_users (fice users uploading 20 files)
• constant_request_rate (2 users uploading 100 files per minute in 5 minutes)

k6 run --env SCENARIO=constant_request_rate upload-availability.js