sv-pubsub

Client for communicating with the Google Pub/Sub service.

Google Pub/Sub is the message broker typically used by microservices to communicate events that occur in one system to the other. These events are typically changes in data, e.g. accounts-update.

The most important concepts in Google Pub/Sub are Topics, Subscriptions and Messages.

Publishing a Message to a Topic without any Subscriptions will result in that message being lost. A Topic can and should be created by both Producers and Consumers of that Topic, to ensure that it exists before either end attempt to interact with it. Subscriptions are created from Topics.

These hold Messages, which Consumers consume via the persistent bidirectional connection it maintains with the Google Pub/Sub service. This is called a "streaming pull", not to be confused with polling/pull, which relies on the Consumer polling for new Messages on an interval.

Publish and Subscribe Architecture

Google Pub/Sub supports several architecture patterns, i.e. configurations of producers, topics, subscriptions and consumers, allowing you to achieve different results:

• Fan in (many-to-one): This setup allows you to make a consumer aware events that may originate in multiple systems. E.g. you have a web app, a mobile app and a partner API that can all trigger an accounts-update event. This is achieved by having many producers publishing to a single topic, the topic is connected to a single subscription, which, in turn, a single consumer will consume events from.
• Fan out (one-to-many): You want a particular event that's been triggered by one or more producers to be published to a single topic. Multiple subscriptions are created from that topic, and each subscription has a copy of the event, which is intended for a different handler. E.g. A user archives an account in a source system, that's broadcasted through a single topic, which is connected to multiple subscriptions. This allows each subscription to be dedicated to a single purpose, allowing consumers of each subscription to perform the necessary updates/cleanup associated with that subscription.
• Load balanced (many-to-many): One or more producers publish an event to a single topic. All consumers should attempt to create a subscription from this topic, then each consumer attaches a handler to this subscription. The Google Pub/Sub service then decides which consumer gets forwarded the message, which the consumer must handle and send an acknowledgement back to the service. Unacknowledged events will eventually time out and be lost. An example of this would be when there's a high throughput of events, and handling them requires multiple consumers.

Authorising with Google Pub/Sub

Authorisation should be done through service accounts. Each microservice typically has a service account. A service account will need permissions to create topics, subscriptions, attach to subscriptions and publish messages in order to authorise.

The credentials required are:

• projectId, e.g. sv-shared-231700
• credentials, i.e. the Application Default Credentials of your Google Service Account

Conventions

Topics should be named after the environment, producer, and event name of the events that pass through it. For example, if a Camber user updates an Account on the live app, we would be broadcasting that events to all Consumers of a Topic named live-camber-accounts-updated.

Subscriptions should be named after the consuming app, e.g. live-auth-live-camber-accounts-updated and set an appropriate retention period. Subscriptions are never automatically cleaned up, however the Messages stored within them are deleted after an hour, by default, although this can be changed.