sfdc-event-hub-common library

This library contains common code used by all Salesforce event hub consumers. Event "enricher" functions that query other services via Lambda direct-invoke or GraphQL are also included in this library.

Making changes

This library is deployed as a private library in NPM. When making changes to the library, you should create beta versions of the library that are published to NPM. You can then test an event consumer (e.g., canary) with the new beta version, and once your changes are verified to be working, you can publish a new version that iterates the semantic version and removes the beta designation.

Publishing to NPM

First, you will need a token to publish to NPM.

1. Search 1Password for "NPM Token - Publish" (in the "Engineering Tools - Dev" vault). Copy the token value.

2. Create a file in your home folder called .npmrc with the following contents:

   //registry.npmjs.org/:_authToken=<copied token value>

You should now be able to execute the make targets to publish the library.

Note on NPM 2FA Authentication: When you run make publish, NPM will require interactive authentication:

1. Open the URL: NPM will provide a one-time authentication URL in the terminal (e.g., https://www.npmjs.com/auth/cli/...).
2. Login: Open the link in your browser and log in with your Guild NPM credentials.
3. One-Time Password (OTP): Use Google Authenticator (or your preferred 2FA app) to provide the required one-time password to complete the publication.

Note on Automatic Tag Detection: The make publish command automatically detects the version type and applies the appropriate NPM tag:

• Prerelease versions (e.g., 0.1.7-beta.0, 0.1.7-beta.1): Published with --tag beta
• Stable versions (e.g., 0.1.7): Published to the latest tag (default)

This ensures prerelease versions don't accidentally become the default install version for consumers.

HOWEVER, FIRST READ THE EXAMPLE WORKFLOW BELOW AND DO STEPS 1&2 FIRST BEFORE MAKE PUBLISH COMMAND!

# View what would be published ("dry run")
$ make publish-dry-run

# Publish to npm
$ make publish

Automated Release Workflow (Recommended)

Use the automated scripts to publish beta and production releases:

Step 1: Publish Beta Versions

# From the monorepo root (sfdc-event-hub-consumers/)
$ cd /path/to/sfdc-event-hub-consumers

# Publish first beta (automatically increments version: 0.1.6 → 0.1.7-beta.0)
$ ./bin/publish-common-beta-update-consumers.sh

What publish-common-beta-update-consumers.sh does:

1. Automatically increments the beta version in sfdc-event-hub-common/package.json
   • If current version is stable (e.g., 0.1.6), creates 0.1.7-beta.0
   • If current version is beta (e.g., 0.1.7-beta.0), increments to 0.1.7-beta.1
2. Runs make publish (pauses for NPM 2FA authentication - see above)
3. Updates all consumer package.json files to use the new beta version
4. Runs npm install to sync lockfiles

After the script completes:

# Review changes
$ git status

# Test consumers with the new beta version
$ cd applications-consumer && npm test

# Commit changes
$ git add .
$ git commit -m "Publish beta v0.1.7-beta.0"
$ git push

# If you need to iterate, run publish-common-beta-update-consumers.sh again (it will increment to beta.1, beta.2, etc.)

Step 2: Publish Production Release

Once your beta version is tested and ready for production:

# From the monorepo root (sfdc-event-hub-consumers/)
$ cd /path/to/sfdc-event-hub-consumers

# Publish production release (removes beta suffix: 0.1.7-beta.0 → 0.1.7)
$ ./bin/publish-common-update-consumers.sh

What publish-common-update-consumers.sh does:

1. Validates that the current version is a beta (e.g., 0.1.7-beta.0)
2. Removes the beta suffix to create the production version (e.g., 0.1.7)
3. Prompts for confirmation before publishing
4. Runs make publish (pauses for NPM 2FA authentication - see above)
5. Updates all consumer package.json files to use the new production version
6. Runs npm install to sync lockfiles

After the script completes:

# Review changes
$ git status

# Test consumers with the production version
$ cd applications-consumer && npm test

# Commit changes
$ git add .
$ git commit -m "Release v0.1.7"
$ git push

# Create a GitHub release/tag
$ git tag v0.1.7
$ git push origin v0.1.7

Manual Publishing Workflow (Step-by-Step for Junior Developers)

This section provides a complete, step-by-step guide for publishing changes to the sfdc-event-hub-common library and updating all consumers in the monorepo.

Step 1: Make Your Code Changes

Make your changes to the library code in sfdc-event-hub-common/src/.

Step 2: Bump the Version

Update the version in sfdc-event-hub-common/package.json:

{
  "name": "@guildeducationinc/sfdc-event-hub-common",
  "version": "X.Y.Z",  // ← Update this version number (e.g., "0.1.15")
  ...
}

Version numbering guidelines:

• For production releases: X.Y.Z (e.g., 0.1.15, 0.1.16, etc.)
• For beta/testing releases: X.Y.Z-beta.N (e.g., 0.1.15-beta.0, 0.1.15-beta.1, etc.)

Step 3: Publish to NPM with make publish

From the sfdc-event-hub-common directory, run:

$ cd sfdc-event-hub-common
$ make publish

What happens during make publish:

1. Syncs package-lock.json with the version in package.json (runs npm install --package-lock-only)
2. Cleans old build artifacts
3. Runs npm run build to compile TypeScript
4. Flattens dist/src/ to dist/ for the NPM package structure
5. Copies package.json and README.md to the dist/ folder
6. Runs npm login to authenticate interactively (see steps below)
7. Runs npm publish ./dist to publish to npmjs.com

Interactive Authentication Steps:

When make publish runs, it will automatically call npm login, which requires two-factor authentication (2FA):

1. Watch the terminal output - NPM will display a message like:

   npm notice Publishing to https://registry.npmjs.org/
   npm notice Please authenticate to publish...
   npm notice Visit this URL to authenticate:
   npm notice https://www.npmjs.com/auth/cli/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

2. Click the browser link - Copy and paste the URL into your browser, or Cmd+Click (Mac) / Ctrl+Click (Windows) if your terminal supports it.

3. Log in to NPM - Use your Guild NPM account credentials (check 1Password if needed).

4. Enter the One-Time Password (OTP) - Open Google Authenticator (or your 2FA app) and enter the 6-digit code for your NPM account.

5. Wait for confirmation - Once authenticated, the terminal will continue and complete the publish. You'll see:

   + @guildeducationinc/sfdc-event-hub-common@<VERSION>

   Example: + @guildeducationinc/[email protected]

Step 4: Update All Consumers to Use the New Version

After successfully publishing, update all consumer applications in the monorepo to use the new version.

From the monorepo root (sfdc-event-hub-consumers/), run the update script:

# Navigate to the monorepo root
$ cd /path/to/sfdc-event-hub-consumers

# Run the update script with the new version
$ ./bin/update-event-hub-common-version.sh -v "<VERSION>"

Generic syntax:

$ ./bin/update-event-hub-common-version.sh -v "<X.Y.Z>"           # For production releases
$ ./bin/update-event-hub-common-version.sh -v "<X.Y.Z-beta.N>"    # For beta releases

Example with specific version:

# Example: Update all consumers to version 0.1.15
$ ./bin/update-event-hub-common-version.sh -v "0.1.15"

# Example: Update all consumers to a beta version
$ ./bin/update-event-hub-common-version.sh -v "0.1.15-beta.0"

What the update script does:

1. Finds all consumer package.json files in the monorepo
2. Updates the @guildeducationinc/sfdc-event-hub-common dependency to the specified version
3. Runs npm install in each consumer directory to update their package-lock.json files
4. Runs npm install at the root to sync the root lockfile

Step 5: Test and Commit

After updating consumers, test and commit your changes:

# Review what changed
$ git status
$ git diff

# Test a consumer to verify the new version works
$ cd applications-consumer
$ npm test

# Commit all changes (library + consumers)
$ git add .
$ git commit -m "Publish sfdc-event-hub-common v<VERSION> and update consumers"
# Example: git commit -m "Publish sfdc-event-hub-common v0.1.15 and update consumers"
$ git push

Complete Example Workflow

Here's a complete example of publishing a new version (using 0.1.15 as an example):

# 1. Edit code in sfdc-event-hub-common/src/
# 2. Update version in sfdc-event-hub-common/package.json to your new version (e.g., "0.1.15")

# 3. Publish to NPM
$ cd sfdc-event-hub-common
$ make publish
# → Click the browser link when prompted
# → Enter Google Authenticator OTP
# → Wait for "✓ Published successfully"

# 4. Update all consumers (replace <VERSION> with your version, e.g., "0.1.15")
$ cd ..  # Back to monorepo root
$ ./bin/update-event-hub-common-version.sh -v "<VERSION>"
# Example: ./bin/update-event-hub-common-version.sh -v "0.1.15"

# 5. Test and commit
$ git status
$ cd applications-consumer && npm test
$ cd ..
$ git add .
$ git commit -m "Publish sfdc-event-hub-common v<VERSION> and update consumers"
$ git push

Manual workflow (Advanced - Legacy Documentation)

Here's an example workflow assuming that changes were being made to the library when version 0.1.0 is deployed in NPM, and all consumers are using that version.

1. Change the version in package.json from 0.1.0 to 0.1.1-beta.0
2. Make code changes, refactor tests, etc.
3. Publish the beta library changes to npm.
4. Test with a consumer (e.g., canary) by changing the dependency version to 0.1.1-beta.0, and running tests or local invoke with the new library version. NOTE: You must run npm install in the consumer directory that uses the new library to get the new beta version, same for new non beta version. You also must run test from NEW terminal window.
5. Iteratively make additional library changes, publish additional betas (-beta2, -beta3, etc.), and retest the consumer until everything works as expected.
6. Now that changes are complete, change the version from 0.1.1-beta.0 to 0.1.1 and publish the new library version to npm.
7. Update the dependency version on all consumers to use 0.1.1.

🗺 The Breadcrumb Trail: Module Map

Navigating a new library can be daunting. Here is the "Breadcrumb Trail" for the common library, mapping every file to its specific role in the pipeline.

⚙️ Configuration & Environment

• jest.config.js: The first point of entry for tests. Sets up the "Early Environment Injection" (STAGE, NODE_ENV) to ensure the config library loads the correct .yaml files.
• jest.setup.js: Global test initialization. Sets up AWS defaults, mock helpers (handleIntegrationTestError), and path resolution for configuration directories.
• tsconfig.json: TypeScript compiler settings. Configured for CommonJS output and strict type checking.

🛡 Core Data Standard (Big6)

• src/big6.ts: The Foundation. Defines the Big6 interface (the 6 required fields) and the GenericEvent interface. Includes the extractBig6() validator used at the start of every pipeline.

🚀 The Processing Pipeline

• src/event.ts: The Orchestrator. Implements the prepareEvents() and postEvents() functions. This is where the 3-stage pipeline (Filter → Enrich → Post) is defined.
• src/sfdc-mapper.ts: The Translator. Converts generic JSON events into the specific CompositeRecord format required by Salesforce Platform Events.
• src/retry.ts: The Shield. Implements the Full Jitter exponential backoff algorithm. Every external call in the library uses this to prevent thundering herds.

Event Processing Pipeline Architecture

The following diagram shows how events flow through the 3-stage pipeline:

flowchart TB
    subgraph Input["📥 Input"]
        KinesisEvent["KinesisStreamEvent<br/>(AWS Lambda event)"]
    end
    
    subgraph Stage1["🔍 Stage 1: Extract & Filter"]
        Extract["extractSupportedEvents()<br/>• Decode Base64<br/>• Parse JSON<br/>• Validate Big6"]
        Filter["Apply Filter Predicate<br/>• meetsBusinessCriteria()<br/>• Registered via registerFilterPredicate()"]
        Extract --> Filter
    end
    
    subgraph Stage2["✨ Stage 2: Enrich (Optional)"]
        Enrich["enrichEvents()<br/>• Call registered enrichers<br/>• Add external data<br/>• Registered via registerEnricher()"]
    end
    
    subgraph Stage3["🏷️ Stage 3: Add Metadata"]
        AddMeta["addMetadata()<br/>• Add metadata.eventType<br/>• Add metadata.eventBusInfo.uuid"]
    end
    
    subgraph Output["📤 Output"]
        PreparedEvents["PreparedEvent[]<br/>(GenericEvent with metadata)"]
    end
    
    KinesisEvent --> Stage1
    Filter -->|"Passed filter"| Stage2
    Filter -.->|"Filtered out"| Skip["⏭️ Skip event"]
    Enrich --> Stage3
    AddMeta --> Output
    
    style Stage1 fill:#81d4fa,stroke:#01579b,stroke-width:2px
    style Stage2 fill:#ffcc80,stroke:#e65100,stroke-width:2px
    style Stage3 fill:#a5d6a7,stroke:#1b5e20,stroke-width:2px
    style Skip fill:#bdbdbd,stroke:#424242,stroke-width:2px

Pipeline Functions:

• prepareEvents(kinesisEvent): Orchestrates all 3 stages
• extractSupportedEvents(kinesisEvent): Stage 1 - Decode and filter
• enrichEvents(events): Stage 2 - Add external data (optional)
• addMetadata(events): Stage 3 - Add standard metadata fields

☁️ AWS Integration Clients

• src/ssm-parameters.ts: Handles runtime feature flags and "Kill Switches".
• src/secrets.ts: Securely fetches credentials (SFDC, Auth0) from AWS Secrets Manager with LRU caching.
• src/dlq.ts: The "Last Resort". Handles the Batch-Level manual DLQ delivery to SQS FIFO.
• src/lambda.ts: A resilient wrapper for cross-lambda invocations with built-in retries.

☁️ External Service Clients

• src/sfdc-api.ts: Salesforce Connectivity. Manages JWT Bearer Auth and executes Composite/SOQL requests using jsforce.
• src/auth0-client.ts: Fetches Machine-to-Machine (M2M) tokens for Guild API access.
• src/graphql-client.ts: A lightweight, native fetch-based client for querying Guild's Internal GraphQL Gateway.

🛠 Tools & Logging

• src/logger.ts: Environment-aware logging. Switches between Powertools (Production/JSON) and GuildLogger (Local/Colorized).
• src/test-helpers.ts: The developer's toolbox for tests. Contains Kinesis event wrappers, fixture loaders, and integration test troubleshooting guides.
• index.ts: The "Barrel Export". Makes all the above utilities available via a single import.

Module Overview

| File | Current Purpose | Future Purpose | | :--- | :--- | :--- | | index.ts | Public API: Barrel export for the library. | Same. | | logger.ts | Standardization: Ensures all consumers log in the same format. | Add Datadog specific tagging. | | auth0-client.ts | Security: Simplifies M2M token retrieval. | Add token caching. | | graphql-client.ts | Enrichment: Standard way to query Guild APIs. | Add automated retries and circuit breaking. | | event.ts | Pipeline: Core logic for filtering, enriching, and posting events. | Support for non-Kinesis event sources. | | sfdc-api.ts | Connectivity: Handles JWT Auth and Composite API execution. | Add support for Salesforce Tooling/Bulk APIs. | | sfdc-mapper.ts | Transformation: Maps generic events to SFDC Platform Event fields. | Support for custom field mapping via config. | | dlq.ts | Resilience: Sends catastrophic failures to SQS FIFO queue. | Add failure analytics/alerting hooks. | | ssm-parameters.ts | Configuration: Fetch runtime kill-switches and feature flags. | Support for dynamic parameter refreshing. | | secrets.ts | Security: Fetch credentials from AWS Secrets Manager. | Add automated secret rotation support. | | big6.ts | Standardization: Validates Guild's required event metadata fields. | Support for Big6 schema versioning. | | retry.ts | Reliability: Centralized exponential backoff configuration. | Add jitter and circuit breaker patterns. | | lambda.ts | Invocation: Client for cross-Lambda calls with retries. | Move to AWS SDK v3 client-lambda best practices. | | test-helpers.ts | Testing: Reusable fixtures, Kinesis wrappers, and error guides. | Add automated performance benchmarking helpers. |

Salesforce Mapper Transformation

The following diagram shows how GenericEvent is transformed to Salesforce format:

flowchart LR
    subgraph Input["📥 Input"]
        GenericEvent["GenericEvent<br/>• Big6 fields<br/>• event_data<br/>• metadata"]
    end
    
    subgraph Transform["🔄 Transformation"]
        direction TB
        GetType["platformEventType()<br/>from SFDC_PLATFORM_EVENT env"]
        ComputeType["computeEventType()<br/>metadata.eventType or Big6"]
        ComputeUuid["computeUuid()<br/>metadata.eventBusInfo.uuid or aggregate_id"]
        Stringify["JSON.stringify(event)<br/>Full event as string"]
        
        GetType --> Build
        ComputeType --> Build
        ComputeUuid --> Build
        Stringify --> Build
        Build["Build CompositeRecord"]
    end
    
    subgraph Output["📤 Output"]
        CompositeRecord["CompositeRecord<br/>• attributes.type<br/>• Event_JSON__c<br/>• Event_Type__c<br/>• Event_UUID__c"]
    end
    
    GenericEvent --> Transform
    Build --> CompositeRecord
    
    style Transform fill:#ffcc80,stroke:#e65100,stroke-width:2px

Field Mappings:

• attributes.type ← SFDC_PLATFORM_EVENT env var (e.g., "Application_Evt__e")
• Event_JSON__c ← Full event serialized as JSON string
• Event_Type__c ← metadata.eventType or constructed from Big6
• Event_UUID__c ← metadata.eventBusInfo.uuid or aggregate_id

The GenericEvent Interface

The library introduces the GenericEvent interface to standardize how events are handled across all consumers.

Benefits:

• IDE Autocomplete: When you type event. in any module, you'll immediately see the 6 required Big6 fields (context_name, entity_type, event_name, etc.).
• Safety: Prevents common typos like contextName vs context_name by enforcing the schema at compile-time.
• Documentation: The code becomes self-documenting, making it clear what an event looks like at a minimum.

import { GenericEvent } from '@guildeducationinc/sfdc-event-hub-common';

function process(event: GenericEvent) {
  console.log(event.context_name); // Autocomplete & Type Safety!
}

Big6 Interface Structure

The following diagram shows the relationship between the core interfaces:

classDiagram
    class Big6 {
        <<interface>>
        +string context_name
        +string entity_type
        +string aggregate_id
        +string entity_id
        +string occurred_at
        +string event_name
    }
    
    class GenericEvent {
        <<interface>>
        +Record~string,any~ event_data?
        +Record~string,any~ metadata?
        +any [key: string]
    }
    
    class CompositeRecord {
        <<interface>>
        +Object attributes
        +string Event_JSON__c
        +string Event_Type__c
        +string Event_UUID__c
    }
    
    Big6 <|-- GenericEvent : extends
    GenericEvent ..> CompositeRecord : transforms to
    
    note for Big6 "Required fields for all Event Hub events\nValidated by extractBig6()"
    note for GenericEvent "Adds optional event_data and metadata\nAllows dynamic fields via index signature"
    note for CompositeRecord "Salesforce Platform Event format\nCreated by makeCompositeRequestBody()"

Key Points:

• Big6: The foundation - 6 required metadata fields
• GenericEvent: Extends Big6 with optional event_data and metadata fields
• CompositeRecord: Salesforce-specific format for Platform Events

Tests

This repo uses Jest as the test runner, and contains both unit and integration tests located in test/unit/ and test/integration/ directories respectively.

Unit Tests - May need to first run command below if you get 'Preset ts-jest not found' error when running make test

$ npm install --save-dev jest

To run unit tests once jest installed:

$ make test

Integration tests

Integration tests must be executed against a specific AWS environment, so you must be authenticated to the target environment with aws sso login. For example, to run the integration tests against the AWS dev environment:

# login via CLI to dev
$ aws sso login --profile=guild-dev

# use aws2-wrap to run the test with AWS dev credentials
$ STAGE=dev aws2-wrap --profile=guild-dev make test.integration

# use aws2-wrap to run the test with AWS staging credentials
$ STAGE=staging aws2-wrap --profile=guild-staging  make test.integration

# developers aren't allowed to run in prod, but the cicd pipeline will do this because it is allowed and if
# integrations tests fail in prod it will let you know when you merge to main

# NOTES:
# If you get a region error, add this to your .zshrc file and uncomment the single hash characters below
# ## AWS values for Guild
# export AWS_REGION=us-west-2
# export AWS_DEFAULT_REGION=us-west-2
# export AWS_DEFAULT_PROFILE=guild-dev