ies-node-api-template

This is a node api template cloud native microservice.

Usage

Installation

To use this library in your react project, ensure the following:

First time setup:

1. Go to Artifactory
2. Set Me Up
3. fxo-cio-ies-npm-virtual
4. Follow the instructions. You can put in your global .npmrc or in project. This acts as a proxy to the npm public repository in addition to our internal repository.

Standard setup:

1. Using your configured package manager, install @ies/node-api-template
2. Import the components as necessary into your app

If using as Backend For Fronted:

1. To enable, add to your environment SERVE_STATIC=true
2. Create a build script that moves the frontend static build to the api/client/build folder where it can be served from.

NOTE: The SERVICE_TYPE envirnment varible is set to BFF by default. This value forces handleNotAuthenticatedResponse middleware to redirect to a defined LOGIN route. Otherwise, this middeware simply returns a HTTP401Error.

Make sure to check for updates to the component library periodically.

Template Components

The library provides a templateServer that can be run with http or https and encapsulates all the common middleware and other setup required (the template currently does not support injecting custom middleware at this time). See the src/start.ts file for example usage.

The primary building block for template consumers is to build their business logic Services around a RestRouter, RestController, and Model , then pass these routes into the templateServer. See the example services/ in this repo. For guidance on service architectural layers, see Restful APIs using MVCS Pattern.

Building Blocks

1. Express a lightweight node http server framework.
2. Express Validator for express middleware validation utilites
3. Class Validator decorator and non-decorate based validation. Used for the model framework in this template.
4. Typescript for strong static typing to make it easier to apply OOP Solid Design Principle and better IDE intellisense
5. Node Fetch A light-weight module that brings window.fetch to Node.js. Standardizes the api with the frontend template!
6. Pino super fast json logger
7. Zipkin for tdistributed tracing
8. Terminus for graceful shutdown and Kubernetes readiness / liveness checks
9. IoRedis full-featured Redis client that is used in the world's biggest online commerce company Alibaba and many other. Used for session storage.
10. GraphQL Apollo implementation of graphql for express
11. Swagger for comment based swagger documentation supporting the open api specification. Makes it easier to embed across project thereby making it easier to remove definitions along with the implementation.

npm scripts

| Command | Description | | ----------------------------- | -------------------------------------------------------------------------- | | start | Starts the app in production mode based off the the static build. | | start:dev | Starts the app in development mode, w/ auto-reload on file changes | | build | Builds the app for production | | build:dev | Builds the app for development | | build:lib | Builds the module to publish for production to the lib folder. | | clean | Cleans out build files and folder(s) | | lint | Lint fixes project src files | | test | Starts tests in CI mode | | test:unit | Starts tests in CI mode running tests with suffix ...unit.test.ts | | test:integration | Starts tests in CI mode running tests with suffix ...unit.integration.ts | | test:watch | Starts tests with file watcher for TDD mode | | test:coverage | Start tests in CI mode with code coverage reporting | | view:coverage | Starts server to serve code coverage report to interact with | | migrate:jest | Operation to attempt an auto update on jest config (new version) | | ----------------------------- | ------------------------------------------------------------------------- |

Contributing

Running in Development

NPM:

start: npm run start:dev

If the hot reload is buggy, run npm run clean && npm run build:dev. It commonly occurs when switching between branches and pulling in new changes.

To avoid using Docker when developing with a frontend when used as a BFF, create a symlink to the client build folder and use two tabs for development mode in both projects.

ln -s <client build path> <api local build path>

Docker:

Start: docker-compose up -d Stop: docker-compose down

:: Notes ::

Redis is currently only for production.

Monitoring redis: docker exec -it redis redis-cli monitor

Troubleshooting - Disable VPN as Redis will throw a security error thinking same host is trying to compromise the box

Expanding Template Capabilities

Anything outside of the services folder is part of the 'template'. The main template file to be aware of is src/templateServer.ts. All components you wish to export as part of the npm module should be exposed in src/index.ts

Once your changes are in, to test your changes you can use npm link locally to test out the project OR you can publish a <packagename>-test version. This should occur on a feature branch as part of the final review process.

Releasing A New Version

Publishing is performed on a release branch. Once you exposed or modified the template framework that is exposed to module consumers,the module version must be updated and the changes logged:

1. Create a release/<VersionNum> branch from dev
2. Update package.json with the version number for the release using npm version <update_type>. Please follow semantic versioning best practices.
3. Update the changelog using feature branch names as entries. Please use best judgement if it needs to be more specific.
4. Commit changes and merge release branch into master branch
5. Run npm publish to publish module to artifactory. This will also kickoff the prepublish script.
6. Alert the team that a new version is pushed

Notes: This will eventually be incorporated eventually in the CI / CD pipeline

Expanding Demo Services

Services are the foundations of building http based microservices with this template grounded on the principles of community NodeJS best practices such as file collocation and a central dedicated error handler.

1. Create a new folder under src/services named for the component you wish to create.
2. Add your router <name>.router.ts file
3. Add your acceptance test <name>.router.test.ts file.
4. Add an index.ts file to export your <name>.router.ts
5. Add your controller <name>.router.controller.ts file
6. Add your model <name>.model.ts file
7. Add other layers you may need like provider(s)...
8. Export your service in src/services/index.ts

Config Design Decisions

Builder pattern

• method that effects template aka. middleware
• method that namespaces common behavior used in consuming apps - not template
• etc.

Testing Practices

Tests not just ensure your code functions as intended but acts as living documentation. The Test Pyramid exposes the trade-offs made between higher level integration tests vs. unit tests.

It is suggested to apply testing as follows:

1. Acceptance tests using Supertest should cover the routes which expose all the endpoints to tap into the apps business logic capabilities. Nock should be used for stubbing outgoing http requests. Feel free to create an env flag to enable and disable nock to go between Integration and E2E level testing.
2. Unit tests using Jest should cover all other cases that dictate business logic as well as data transformations and validation.

Restful APIs using MVCS Pattern

This pattern is commonly found in enterprise application architecture and popularized by SpringBoot. It contains the following component layer types in bottom up order:

Model - comprises interface types and other abstract definitions that help model the business domain.

Example:

DAO / DAL - An abstraction that your service layer can call to get/update the data it needs. This layer is data-centric and will either call a Database or some other system (eg: LDAP server, web service, or NoSql-type DB). A few common libraries that provide these capabilities are:

1. Relational DB ORM: sequalize
2. Mongo DB ORM (NoSQL): mongoose

You can use your models to cast the schemas and data access patterns in these libraries to conform to a independent implementation. Another variation of this is known as a 'Provider'.

Example: TBD

Provider - encapsulates the interaction with external data providers aka. 3rd party APIs and other microservices. This component is preferred over DAOs / DAL for read-only access.

Example: TBD

Repository - designed to separate your domain objects from data access logic (DAO/DAL) by acting as a domain specific Collection (aka. higher level interface). It can use DAO to restore the business objects that your application / API is responsibile for.

Example: TBD

Controller - orchestrates the calls between the api layer and the service layer. A controller can call more than one service but only a single service call is demonstrated in the demo app. Usually if your controller is calling many services its one indication to use a different solution like GraphQL.

Example: TBD

Services - encapsulates business logic as finalized output to be called by a controller - a layer commonly found in enterprise architecture. In this template we further suggest that each named service directory in the services/ folder acts as a internal "microservice" that can be more easily refactored out into their own service in the future.

Example: TBD

View - option to serve static content such as a build of a frontend application to enable a BFF pattern

Example: TBD

GraphQL Patterns

Follows the domain driven folder structure of services. It is reccommended to separate your .typedefs.ts, .resolvers.ts, .mutators.ts, etc into their own file.

Benefits:

• Modular; a domain can be externalized in its own module
• Scalable; this structure adapt quite well to very large apps
• Clean; forcing a sub-domain into a parent make sure there is no overlap

Cons:

• Overkill for smaller apps
• Conflicts can arise for establishing ownership of sub-domains (however resolving them is a really good thing)
• Domain boundaries might not be clear from the start

Error Handling

The template implements the most critical parts of Error Handling Practices. It covers catching uncaughtException and unhandledRejection that can put your app into an undefined state as well as distinguishing between client and server errors. It handles catching both sync and async errors within the middleware, allowing us to free our controllers’ code with error handling.

We want to throw an exception and make sure our dedicated middleware will handle it for us. This template creates a dedicated ErrorHandler class available for unit-testing.

We have three different error handlers for each use case:

1. handle404Error the way you handle 404 in express. By adding a fallback middleware if nothing else was found.
2. handleClientErrors catches client API errors like Bad request or Unauthorized.
3. handleServerErrors a place where we handle “Internal Server Error”.

Authentication Middleware

All of the template's router-level authentication middleware verify an authenticated request using Passport's Request interface. A request is authenticated when the isAuthenticated method from the Request instance returns true.

There are five different authentication middleware within the template, each respond differently to a non authenticated request:

1. ensureAuthenticated if request is not authenticated, request will be redirected to AUTH_ROUTES.LOGIN.
2. ensureSessionAuthRedirectUrl if request is not authenticated, request will be redirected to the requests originalUrl.
3. ensureAuthUser if request is not authenticated, HTTP401Error will be thrown.
4. requirePolicy if request is not authenticated, the request will be redirected to AUTH_ROUTES.LOGIN allow the user to login. Additionally, if the user making the request has not accepted the applications policy, validateUserWithServiceIdApi, the request will be redirected to the req.requestUrl.
5. ensureAdminUser if a requesting user does not have the configured admin blue group a HTTP403Error will be thrown.

Session Managment

The app uses redis for its store because express-session default in memory store is not designed for production use. Due to many circulating issues with the reccomended redis client, Alibaba and other top companies support a more performant and powerful one used in this project - ioredis.

For development you can use docker to spin up a local instance or you can setup tls by applying the cert in the env under redisCertFileName to connect to the IBM IES Team's Cloud hosted instance.

Custom Configuration

This templates configuration can be modified to fit the needs of the consuming application. Configuration must be modified within the start.ts file just before the server.start() invocation.

Below are methods that can be called on the ApplicationConfig class, modifying the application configuration:

• addBluegroups - adds values to the bluegroups property of the application's baseConfig.
• usePolicies - allows developer to configure a privicy policy for an application.
• apply - adds specified property to the application's baseConfig.
• useServices - adds property api to the application's baseConfig.
• setBasicAuth - adds property basicAuth to the application's baseConfig.
• enableMaintenanceMode - adds basicAuth to the application's baseConfig. Also adds common with properties serviceId and servicesApi to baseConfig. Does not overwrite information set in common by usePolicies.

Implementing Swagger

You can use swagger to document your REST routes from this template. To do so, simply pass in a swagger configuration object when instantiating the server from the IESTemplateServer constructor. After passing in the swagger object, go ahead and start documenting your routes with comments so that swagger-jsdoc can pick it up! For details on what that configuration object should look like, or how to document your routes, please refer to the examples provided in the code.

Setting up an oAuth flow in Swagger can be a little tricky, and this template aims to make that a little easier for you. To add oAuth flow:

• Provide a valid clientId and clientSecret via the env variables OPENID_CLIENT_ID and OPENID_CLIENT_SECRET

• Authorize provisioner to redirect to your application. In addition to the normal authorizations, you will need to add the special routes /oauth2-redirect.html* and /oauth2-redirect.html attached to the end of your base-url for auth flow to work. For example, your application may have the following in provisioner:

  • https://my-sample-app.prod.identity-services.intranet.ibm.com/oauth2-redirect.html*
  • https://my-sample-app.prod.identity-services.intranet.ibm.com/oauth2-redirect.html
  • https://my-sample-app.prod.identity-services.intranet.ibm.com

  Note that your provisioner setup may differ than this. A helpful hint is that if you successfully authenticate with the authorization server, but can't redirect back to your application because of an error telling you that the redirect url is not valid, you probably haven't whitelisted the right url in provisioner. Try looking at where provisioner is trying to rediret you in the url to help troubleshoot this further.

• Create a security scheme for oAuth. The name can be arbitrary, but the type must be oauth2

  • Provide a valid authorizationUrl inside the authorizationCode property. This is the url that swagger will attempt to authenticate against
  • Set token url to /api/proxy/oidc/endpoint/default/token. When the authorizationUrl redirects successfully, it will send a nonce back to swagger-ui, which will then be redireted to the server. The server will then forward this request to the JWT server with the nonce and client_secret and receive a JWT back. (The actual flow is slightly more complicated, with the redirect first hitting a special html file before redirecting to the swagger-ui)

  Here's what our securityScheme looks like at this point:

  const securitySchemes: SecuritySchemes = {
    oAuth: {
      type: "oauth2",
      description: "OAUTH2 Login Flow",
      flows: {
        authorizationCode: {
          authorizationUrl:
            "https://preprod.login.w3.ibm.com/oidc/endpoint/default/authorize",
          tokenUrl: "/api/proxy/oidc/endpoint/default/token",
          scopes: { openid: "root access" }, // This can be omitted, only necessary if you want to apply scopes
        },
      },
    },
  };

• Under swaggerUIOptions, add an oauth property with a clientId property that points to the appropriate clientId

  And this is what our swaggerUIOptions object looks like at this step:

  const swaggerUIOptions = {
    swaggerOptions: {
      oauth: { clientId: getStringEnvVar("OPENID_CLIENT_ID", "") }, // clientId is picked up via a utility function which reads it from our env variables
    },
    explorer: true,
  };

• Import the ProxyRoutes object from the template somewhere in your project where you can add it to your base routes object. A common pattern that the team uses is to add this route in the index file under the services folder.

That's it! Congratulations on making it this far, and now how you consume this login flow with swagger-jsdoc is up to you. Its likely though that you'll want to add this flow everywhere.

Using comments for swagger-jsdoc, apply your security:

/**
 * @swagger
 *  security:
 *    - oAuth: []
 */

In this example, oAuth references the arbitrary name that was specified earlier in our securitySchemes. If you named it something else like myLoginFlow, you would replace oAuth with myLoginFlow. Our examples don't implement scope, but you should be able to add it to your securityScheme if desired.

To implement security at a route level, we just add the security property to our comments for swagger-jsdoc to pick up:

/**
 * ...more comments above
 *   security:
 *    - oAuth: []
 * ...more comments below
 */

You can override your security on a route with a - [] under security, opening up the api route to everyone. Further reading

Please refer to the implementation in this project for a reference point if you get stuck or confused. As a friendly reminder, Swagger is based off of the OpenAPI specification.

Running in Maintenance Mode

This template offers the ability to run an application in maintenance mode. In this feature, the applications maintenanceStatus will be checked by common-services before determining whether or not to serve the file index.html or maintenance.html. To enable this mode:

• Ensure that the env variable serveStatic is set to true
• Add the following env variables to your env file
  • SERVE_STATIC set to true
  • ENABLE_MAINTENANCE_MODE set to true
  • A variable to track the services API (e.g. SERVICES_API=https://ies-common-services-api.dev.identity-services.intranet.ibm.com/api/v1)
  • A variable to track the id of the app in common services (e.g.SERVICE_ID=ies-hsk-procurement)
  • Two variables to track basic auth (e.g. BASIC_AUTH_USER=username BASIC_AUTH_PASS=pass)
• Apply your config by invoking the enableMaintenanceMode method off of ApplicationConfig
• Ensure that the consuming applications public folder possesses both an index.html and maintenance.html file

The consuming application decides how to structure their maintenance.html file. However, an example has been provided as part of this template. The example provided will allow a user to show the login button by hitting the ENTER key five times. If an admin user logs in at this point, they will be able to see the application as normal. Regular users will still see the maintenance page.

Site Notices

Site notices are messages that developers want the application users to see. A notice will be returned from the api, api/site-notice endpoint, if the notice's start and end date fall between the current day.

To configure site notices

1. The consuming application must be on node template version 3.0.0 or greater

2. The projuct must contain a DASHBOARD_SERVICE_API environment variable that points to https://ies-dashboard-service.TARGET ENV.identity-services.intranet.ibm.com/api/v1

3. Within /api/src create a file titled applyConfig.js (if this file does not exist already) and use the useDashboardNotice method on the ApplicationConfig object to specify which offering to pull notices from. Below is an example apply Config.ts file.

import { ApplicationConfig, getStringEnvVar } from "@ies/node-api-template";

// note that this implementaion uses getStringEnvVar to get the value of TARGET_OFFERING_ID from process.env
ApplicationConfig.useDashboardNotice(getStringEnvVar("TARGET_OFFERING_ID"));

The TARGET_OFFERING_ID is the offering id of the offering that your application will receive notices about.

4. Lasly, within /api/src/services create a file index.ts (if this file does not exisit already), import the SiteNoticeRoutes from the node api template and add it to your exported routes. Below is an example index.ts file.

import { RestRouter, SiteNoticeRoutes } from "@ies/node-api-template";

import { UserRoutes } from "./authUser"; // example project specific route

const routes: RestRouter[] = [UserRoutes, SiteNoticeRoutes];

const getRoutes = async (): Promise<RestRouter[]> => {
  return routes;
};

export { getRoutes };

Future Ideas

Provide a more expressive framework to dynamically generate routes and swagger similar to SpringBoot annotations. There are currently two popular npm modules for this: TSOA and Typescript-Rest. The team has explored both frameworks in the following branches:

TSOA Branch TypescriptRest Branch

The biggest hurdle in both frameworks was the lack of support and/or documentation for custom security middleware. TypescriptRest seems to have the most promise with PassportJs support as well as security middleware injection which is encouraging in order to try again in the future.

Add GraphQL...