@visualvault/vv-vertical-event-logger

Private NPM package that provides Express middleware for tracking user activity events across VisualVault Vertical Solutions. Events are written to a dedicated MySQL database for internal product usage analytics and customer segmentation. No PII is captured — only system identifiers.

How it works

1. Middleware registration — createEventLoggerMiddleware() returns a standard Express middleware function that is registered once in the host app via app.use().

2. Event detection — For each incoming request, the middleware runs all registered event handlers — the built-in ones plus any the host app supplied via options.handlers (see Logging your own event types). Each handler decides whether that request is relevant (e.g., a login POST) and, if so, attaches a listener to the response finish event to capture the outcome after the response is sent.

3. Buffering — Captured events are held in an in-memory buffer. The buffer flushes to the database when either:

   • The buffer reaches the configured batch size (EVENT_LOG_BATCH_SIZE, default 50)
   • The flush interval elapses (EVENT_LOG_FLUSH_INTERVAL_MS, default 30000 ms)
   • The process receives SIGTERM or SIGINT, or the host app calls stop() (see Shutdown)

4. Database write — Flushed batches are written to the app_events table in the dedicated logging database using Sequelize bulkCreate. A batch is removed from the buffer only after the write is confirmed, so a transient failure (network blip, DB restart, pool exhaustion, deadlock) costs a retry on the next cycle rather than the batch. Events are discarded in only two cases:

   • The buffer reaches EVENT_LOG_MAX_BUFFER_SIZE (default 10000) — the newest events are shed, bounding memory if the database stays down.
   • The database rejects the batch for a reason retrying cannot fix (an oversized or malformed column value). Retrying such a batch forever would park it at the head of the buffer and stop all logging, so it is dropped and logged.

5. Schema migration — At startup the package connects to the logging database and applies its own migrations (via Umzug) to bring the schema up to date. This is non-fatal: if the logging database is unreachable, the failure is logged, events keep buffering, and init is retried on the next flush. The host app always starts normally.

Adding new event types

A host app adds its own event types via options.handlers — no fork of this package or upstream PR required. See Logging your own event types for the full host-facing walkthrough.

Contributing a new built-in event type to this package itself follows the same shape, just registered in the package's own EVENT_HANDLERS array instead of passed in as an option:

1. Create src/handlers/<eventType>Handler.js — implement the attach(req, res, buffer, config) interface (see loginHandler.js for the reference implementation).
2. Register it in the EVENT_HANDLERS array at the top of src/middleware.js.
3. Call config.logEvent({ actionType, eventData, userId, req }) to capture the event — never construct/push the row directly. action_type is free text (VARCHAR(100)), so a new event type generally needs no new migration just to introduce its value; a migration is only needed if the new type needs new columns (see Changing the schema).

Database schema

The package writes to a single table, app_events, in a dedicated logging database. The schema is owned by this package and applied automatically at startup by the migrations in src/db/migrations/. The equivalent DDL, for review:

CREATE TABLE app_events (
  id                   BIGINT UNSIGNED AUTO_INCREMENT  NOT NULL,
  app_id               VARCHAR(10)                     NOT NULL  COMMENT 'Vertical app identifier, e.g. LP, LDA',
  user_id              VARCHAR(255)                    NOT NULL  COMMENT 'Internal VV user ID or userDatabaseId — no PII',
  action_type          VARCHAR(100)                    NOT NULL  COMMENT 'Event type — free text, no migration needed for new values',
  event_data           TEXT                            NOT NULL  COMMENT 'Serialized JSON event payload (shape is event-type-specific)',
  customer_id          VARCHAR(255)                        NULL  COMMENT 'Tenant identifier for customer segmentation',
  customer_database_id VARCHAR(36)                         NULL  COMMENT 'Tenant database identifier for customer segmentation',
  timestamp            DATETIME                        NOT NULL  COMMENT 'UTC time the event occurred',
  environment          VARCHAR(50)                     NOT NULL  COMMENT 'NODE_ENV value at time of event',
  created_at           DATETIME                        NOT NULL  DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (id),
  INDEX idx_app_env         (app_id, environment),
  INDEX idx_user_id         (user_id),
  INDEX idx_action_type     (action_type),
  INDEX idx_timestamp       (timestamp),
  INDEX idx_customer_id     (customer_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

Migration history is tracked in a second table, event_log_migrations, created and maintained automatically. It is named distinctly from Umzug's default (SequelizeMeta) so this package's history can never collide with a host app's.

Changing the schema

Never edit an applied migration — databases that already ran it will not re-run it. Instead:

1. Add src/db/migrations/<nnn>-<description>.js, exporting { name, up({ context }), down({ context }) } where context is the Sequelize QueryInterface.
2. Append it to the MIGRATIONS array in src/db/migrations/index.js.
3. Mirror the change in src/db/models/AppEvent.js and in the DDL above.
4. Publish a new package version. Each host app picks the change up on its next deploy, when the migration runs at boot.

Upgrading from V1: V1 of this package created app_events with sync(), which never added the indexes above. The first migration detects an existing table, skips the create, and adds only the missing indexes — so it is safe against both a fresh and a V1 database.

Upgrading from 1.x: 2.0.0 generalizes action_type (ENUM('login') → VARCHAR(100), free text) and renames event_url to event_data (VARCHAR(2048) → TEXT, holding serialized JSON instead of a bare URL path). Existing event_url values are automatically wrapped as {"url": "<value>"} so every row holds valid JSON afterward — no manual data fixup needed. This is a breaking change for anything querying event_url directly or relying on action_type being a closed enum; update those before upgrading.

Integrating into a Vertical app

1. Set up the logging database

Create a dedicated MySQL database in each environment (local, dev, preprod, prod). The package creates and migrates its own tables at startup, but the database itself must exist beforehand.

Create a dedicated database user for the package. Because the package runs migrations at boot, that user needs DDL privileges in addition to the runtime INSERT/SELECT:

GRANT SELECT, INSERT, CREATE, ALTER, INDEX ON vv_event_log.* TO 'logging_user'@'%';

| Privilege | Needed for | |---|---| | INSERT | writing event rows (runtime) | | SELECT | reading migration history (boot) and dashboard queries | | CREATE | creating app_events and event_log_migrations on first boot | | ALTER, INDEX | applying later schema changes and indexes |

If your infrastructure policy forbids granting DDL to an application user, apply the migrations out-of-band with a privileged user and grant the runtime user only SELECT, INSERT. The app will then log a migration failure at boot if the schema is ever behind — it will not crash, but events will not be written until the schema is current.

2. Install the package

Publish to the VisualVault private NPM registry, then install in the target app:

npm install @visualvault/vv-vertical-event-logger

If the private registry requires scoped configuration, add the following to the app's .npmrc:

@visualvault:registry=https://<vv-private-registry-url>/

3. Add environment variables

Add the following variables to each config.*.env file in the host app and encrypt them with dotenvx as appropriate. All EVENT_LOG_* variables are read directly from process.env — the host app's existing env loading handles them.

| Variable | Required | Default | Description | |---|---|---|---| | EVENT_LOG_DB_HOST | Yes | — | Hostname of the logging database | | EVENT_LOG_DB_PORT | No | 3306 | Port of the logging database | | EVENT_LOG_DB_USER | Yes | — | Database user | | EVENT_LOG_DB_PASSWORD | Yes | — | Database password | | EVENT_LOG_DB_NAME | Yes | — | Database name (e.g., vv_event_log) | | EVENT_LOG_APP_ID | Yes | UNKNOWN | Short identifier for this vertical app (e.g., LP, LDA) | | EVENT_LOG_DEFAULT_CUST_ID | No | null | Default customer identifier written to every event row | | EVENT_LOG_DEFAULT_CUST_DB_ID | No | null | Default customer database identifier written to every event row | | EVENT_LOG_BATCH_SIZE | No | 50 | Number of events that triggers an immediate flush | | EVENT_LOG_FLUSH_INTERVAL_MS | No | 30000 | Milliseconds between automatic flushes | | EVENT_LOG_MAX_BUFFER_SIZE | No | 10000 | Maximum events held in memory while the database is unreachable. Beyond this, the newest events are dropped | | EVENT_LOG_LOGIN_PATH | No | /login | POST path used for login — override if the app uses a different route | | EVENT_LOG_HANDLE_SIGNALS | No | true | Set false if the host app has its own shutdown sequence and will call stop() itself (see Shutdown) |

The numeric variables must be positive integers (1e5 is accepted and means 100000). Anything else — a non-numeric value, a fractional one, or a digit-separated one like 30_000 — falls back to the default with no error, so check the boot logs after changing them. EVENT_LOG_HANDLE_SIGNALS accepts false, 0, no and off in any case; every other value is true.

Example additions to config.local.env:

# Event Logger
EVENT_LOG_DB_HOST=localhost
EVENT_LOG_DB_PORT=3306
EVENT_LOG_DB_USER=logging_user
EVENT_LOG_DB_PASSWORD=your_password_here
EVENT_LOG_DB_NAME=vv_event_log
EVENT_LOG_APP_ID=LP
EVENT_LOG_DEFAULT_CUST_ID=
EVENT_LOG_DEFAULT_CUST_DB_ID=

4. Register the middleware

In the host app's app.js, require and register the middleware after session and Passport middleware (the login handler reads from req.session):

const { createEventLoggerMiddleware } = require('@visualvault/vv-vertical-event-logger');

// ... existing middleware (cors, session, passport, etc.) ...

app.use(createEventLoggerMiddleware());

// ... route definitions ...

That's the complete setup. No additional code changes are needed in the host app.

Multi-tenant hosts: resolving customer_id per request

EVENT_LOG_DEFAULT_CUST_ID/EVENT_LOG_DEFAULT_CUST_DB_ID are read once at startup, so they work as-is for a host that serves a single tenant per deployment. A host where one running instance serves many tenants (distinguished per-request, e.g. via req.session) needs a different value on every event, not one static default for all of them. Pass resolveTenant to get that:

const eventLogger = createEventLoggerMiddleware({
  resolveTenant: (req) => ({
    customerId: req.session?.customerId,
    customerDatabaseId: req.session?.customerDatabaseId,
  }),
});
app.use(eventLogger);

resolveTenant(req) is called once per captured event, from the same guarded context as the rest of event capture — a throw is caught and logged, never surfaced to the request. Return { customerId, customerDatabaseId }; either field left undefined/null falls back to the corresponding EVENT_LOG_DEFAULT_CUST_* value. Omit the option entirely and behavior is unchanged from before it existed — this is purely additive.

Logging your own event types

A host app adds a new event type by writing a handler module and passing it via options.handlers — the same attach(req, res, buffer, config) interface loginHandler.js itself implements (see that file for the reference example). No fork of this package or upstream PR is required; action_type is free text, so a new value needs no schema change either.

// myAppEventHandler.js
function attach(req, res, buffer, config) {
  if (req.method !== 'POST' || req.path !== '/widgets') return;

  res.once('finish', () => {
    // Guard this callback yourself — see the note below. logEvent() itself
    // never throws (see below), but your own code here still can.
    try {
      const result = config.logEvent({
        actionType: 'widget_created',
        eventData: { widgetId: req.body.id },
        userId: req.session?.userId || 'unknown',
        req, // optional — passed through to resolveTenant, if configured
      });
      if (!result.ok) {
        console.error('widget handler passed invalid input to logEvent():', result.error.message);
      }
    } catch (err) {
      console.error('widget handler failed to capture event:', err.message);
    }
  });
}

module.exports = { attach };

const eventLogger = createEventLoggerMiddleware({
  handlers: [require('./myAppEventHandler')],
});
app.use(eventLogger);

Your handler decides when to log (request matching, response-lifecycle timing) and pulls whatever request-specific data your event needs (userId, and anything else you want to fold into eventData); config.logEvent(...) owns actually writing the row — tenant resolution (via resolveTenant, if configured, automatically — no need to duplicate that logic), JSON serialization, and validation, so every handler (built-in or yours) gets that consistently. actionType and eventData are always required; userId is too, unless your own logic can supply one.

logEvent() never throws — it returns { ok: true } or { ok: false, error }. Missing actionType/eventData/userId comes back as { ok: false, error } rather than an exception, so a caller that forgets to check it can never be crashed by it — but a caller that does check it gets a precise, synchronous reason for what it passed wrong. { ok: true } means the event was accepted into the in-memory buffer, not that it has been written to the database yet — that happens later, in a batch with other events, so there is no return value that can confirm eventual delivery (see Behavior notes).

Still guard your own response-lifecycle callback in try/catch, exactly as the example above and loginHandler.js both do — not because of logEvent() itself, but because of whatever else your callback does (reading req.session, for instance, the way loginHandler.js's own comment warns about). The middleware's own per-request guard (in src/middleware.js) only wraps the synchronous attach() call itself — anything you defer into res.once('finish', ...) runs on a later tick, outside that guard, and an uncaught throw there can still crash the host process.

Don't have a request to hook into at all — a background job, a scheduled task failure? Call the same primitive directly instead:

const eventLogger = createEventLoggerMiddleware();
app.use(eventLogger);

// later, from anywhere — no req available:
eventLogger.logEvent({
  actionType: 'nightly_sync_failed',
  eventData: { reason: err.message },
  userId: 'system',
});

req is simply omitted — resolveTenant (if configured) is not consulted, and customer_id/customer_database_id fall back to the static EVENT_LOG_DEFAULT_CUST_* values (or stay null).

Shutdown

By default the package registers its own SIGTERM/SIGINT handlers, which drain the buffer, close the database pool, and then exit — so no host-app code is required.

If the host app already has its own shutdown sequence, set handleSignals: false and let it own shutdown. Otherwise both handlers run, and whichever calls process.exit() first cuts the other short — this package's drain typically finishes first and would kill the host's in-flight requests:

const eventLogger = createEventLoggerMiddleware({ handleSignals: false });
app.use(eventLogger);

// in the host app's existing shutdown path, before it exits:
await eventLogger.stop();

stop() clears the flush timer, drains the buffer, and closes the Sequelize pool — the pool matters, because its idle-eviction timer would otherwise keep the process alive for up to 10s after the app is otherwise done. It is idempotent, safe to call more than once, and terminal: the logger stops accepting events once it resolves. It resolves to { pending, discarded } — pending is non-zero only if the logging database was unreachable during the drain, and both are logged automatically.

The middleware also exposes eventLogger.ready, a promise that resolves once the boot-time database init has settled — awaiting it is optional and it never rejects.

Behavior notes

• All environments are logged. Events are written regardless of NODE_ENV. Use the environment column to filter non-production rows in dashboards.
• Failures are non-fatal. If the logging database is unreachable, the host app still starts and serves normally. Events buffer in memory and both the connection and the schema migration are retried on each flush cycle. Nothing in the package can turn a host app request into a 500 — handler attachment and event capture are both guarded.
• Failed batches are retried, not dropped. A batch is spliced out of the buffer only after the write is confirmed. Discards are limited to the two cases listed under How it works and are always logged. Buffered events do not survive a process crash.
• Timestamps are UTC. The database connection is pinned to +00:00, so timestamp and created_at are UTC regardless of the app server's or MySQL server's timezone.
• event_data holds a serialized JSON payload, shaped per event type. loginHandler.js writes {"url": "<path>"} — the request path without the query string, which is deliberately excluded because it can carry tokens or email addresses. This table holds no PII, and that is not enforced by the schema — a VARCHAR(2048) URL-only column left little room for PII almost by accident; an open JSON payload does not have that natural ceiling. Any handler writing event_data must keep it PII-free by convention. The path in loginHandler.js's payload is the full mounted path, so a logger mounted on a sub-router still records which route produced the event.
• Concurrent instances are safe. Several app instances booting at once against one logging database will not corrupt the schema: whichever loses the migration race detects it and converges on a second pass.
• The customer_id and customer_database_id fields default to the EVENT_LOG_DEFAULT_CUST_ID/EVENT_LOG_DEFAULT_CUST_DB_ID env vars, read once at startup — correct for a host serving one tenant per deployment. A multi-tenant host should pass resolveTenant (see Multi-tenant hosts) to populate these per request instead.
• logEvent() never throws — invalid input is reported via its return value, not an exception. Missing actionType, eventData, or userId comes back as { ok: false, error }, consistent with this package's promise that nothing in it can crash the host process. { ok: true } confirms the event was accepted into the buffer only — not that it has been written to the database, which happens later in a batch with other events. See Logging your own event types.