AC UserActionLog Connector

Connect an app to our userActionLog service with this module.

The module receives some log payload from backend applications, sends them to AWS firehose for processing/converting and stores the data (as original as well as converted) to S3.

Breaking change in version 4 / Upgrading from version 3

• improved compatibility with AWS SDK
• use AWS_PROFILE instead of custom profile
• let SDK handle session management (refresh tokens)
• function is now a class - so initializing slightly changes

To upgrade (your local or non EC2 machines), please read the section "Local development/non EC2 instances"

Breaking change in version 3

• works with Node16 or higher
• async/await - no callback!
• uses AWS IAM roles or AWS IAM profiles instead of IAM credentials

Upgrading from version 1 to 2

Please note that version 1 is no longer supported. Version 2 uses AWS Kinesis Firehose, AWS Glue and AWS Athena.

The data is stored in S3 and you can export/download it from there any time.

Usage

// Init during bootstrap

const ualConnector = require('ac-useractionlog-connector')

ualConnector = new ual({
  // optional config parameters (see below)
})

// now send logs like this
await ualConnector.log({ ip, processingTime, statusCode, logMeta: { body: req.body } }, req)

Configuration options

|Parameter|Default value|Description| |-----|----|----| |region|eu-central-1|Region for Firehose| |deliveryStreamName|UserActionLogs|Name of the stream in Firehose| |applicationName|AC-UAL-AppNameNotSet|Name of the logging application| |applicationVersion|AppVersionNotSet|Version of the logging application|

AWS Credentials

Credentials are fetched using the approach described here: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-credentials-node.html

Local development/non EC2 instances

AWS SDK automatically handles expired session tokens if you follow these instructions:

1 Download this script -> https://github.com/AdmiralCloud/bashHelper/blob/master/awsConnect/getRoleSession.sh 2 Prepare the file ~/.aws/config and add a configuration for the role (e.g. ac-api-instance-role) you want to use:

// dev is the profile name of the IAM user (your IAM user) that is allowed to assume the role. 
// The profile must be defined in ~/.aws/credentials

[profile ac-api-instance-role]
credential_process = /ABSOLUTE_PATH/BashHelpers/awsConnect/getRoleSession.sh  "ac-api-instance-role" "dev"

3 export AWS_PROFILE=role in the project where this connector is used. You can also add it to pm2 config json. 4 Run the script in your project that uses this connector. The SDK will handle everything else for you!

Preparation

Glue

Create a manual table in Glue:

Name it "UAL-Default", select the database "useractionlog" (or create it if not existing), choose Type "Kinesis" with Location "UserActionLog" and the Kinesis URL "https://glue.eu-central-1.amazonaws.com". Select data format JSON. And then use the following fields:

| Pos | Field | Type | | --- | --- | --- | 1 | ip | string | 2 | method | string | 3 | controller | string | 4 | action | string | 5 | statuscode | int | 6 | processingtime | int | 7 | userId | int | 8 | customerId | int | 9 | mediaContainerId | int | 10 | created | bigint | 11 | createdAt | timestamp | 12 | platform | string | 13 | platformVersion | string | 14 | clientVersion | string | 15 | token | string | 16 | clientid | string | 17 | deviceidentifier | string | 18 | useragent | string | 19 | payload | string | 20 | truncated | boolean | 21 | response | string |

AWS Firehose

First you have to create a delivery stream (Firehose) in AWS Kinesis. Name it "UserActionLogs", select "Direct PUT or other sources".

On the next page, choose "Convert record format" and select "Apache Parquet" format. Use the table definition from Glue (above).

The destination should be S3 logbucket with prefix "ual/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/". The error prefixx can be "ual_errors/".

Also select S3 backup and use the logbucket and prefix "userActionLogs.

Finally, create the delivery stream.

Analysis

To analyze your data you have to create a Athena database and query it.

Create the table

Before running the query below, consider the following:

• If you don't need the whole month, but only a day, remove "day" from partition and update the location with the day (day=xx)
• The fewer data is indexed, the faster the process.

// Create the table in database useractionlog
CREATE EXTERNAL TABLE IF NOT EXISTS ual_202011 (
  ip string,
  method string,
  controller string,
  action string,
  statusCode int,
  processingtime int,
  userId int,
  customerId int,
  mediaContainerId int,
  createdAt timestamp,
  platform string,
  clientVersion string,
  payload string,
  token string,
  clientId string,
  deviceIdentifier string,
  response boolean,
  response string      
  userAgent string,
  created bigint,
) 
PARTITIONED BY (
  day string,
  hour string
)
STORED AS PARQUET 
LOCATION 's3://logs.admiralcloud.live.fra/ual/year=2020/month=11/'

After creation and before querying, you need to load the partition using the following command:

MSCK REPAIR TABLE ual_202011;

When querying data, please consider reducing the data to parse using the partions created above

// Example Query using partitions

SELECT * FROM "useractionlog"."ual_202011" 
where hour >= '14' AND hour < '15'
limit 10;

Error handling

If creating a Glue table is not working via AWS console, you can use CLI but have to edit/add field definitions etc later. So use this only as fallback!

// FALLBACK
  aws glue create-table \
  --database-name useractionlog \
  --table-input  '{
    "Name":"ual-default", "StorageDescriptor":{}}' \
  --endpoint https://glue.eu-central-1.amazonaws.com

Links

• Website
• Twitter (@admiralcloud)
• Facebook

License

Copyright mmpro