@backtrace/forensics

v0.7.6

Published

2 months ago

Simplifies making and parsing queries to and from Coroner.

Downloads

0High
0Medium
0Low

backtrace-forensic

This is a library aimed to simplify Coroner querying in Typescript/Javascript.

Building

npm install
npm run build

Running tests

npm run test

Usage

Creating a query

You can create a query in two ways:

Create a Forensics instance and create queries from there:

import { Forensics } from '@backtrace/forensics';
const instance = new Forensics({
    /* any options here */
});
const query = instance.create();

Use the static function from Forensics:

import { Forensics } from '@backtrace/forensics';
const query = Forensics.create({
    /* any options here */
});

Options

You can provide options to change the behavior of the library:

defaultSource

Data from this source will be used as defaults for API calls.

You can override this in post() functions in queries.

Example

options.defaultSource = {
    address: 'http://sample.sp.backtrace.io',
    token: '00112233445566778899AABBCCDDEEFF',
};

// Will use `address` and `token` from defaults
query.post({ project: 'coroner' });

apiCaller

Use this to override the default API caller for API calls.

The caller must implement the ICoronerApiCaller interface.

Example

options.apiCaller = new NodeCoronerApiCaller({
    disableSslVerification: true,
});
options.apiCaller = async () => new MyCoronerApiCaller();
options.apiCaller = {
    async create() {
        return new MyCoronerApiCaller();
    },
};

plugins
Register any plugins within this instance or query. See Plugins for more info.
Example
```
options.plugins = [myPlugin({ option: true })];
```

Using the query functions

To build a query, chain the function calls until your query is in desired state:

query.filter('attribute', 'equal', 'abc').fold('attribute', 'head').fold('attribute', 'tail').group('fingerprint');

Each call creates a new query with cloned request, so it is possible to modify a query without changing the parent:

const query = instance.create().filter('attribute', 'equal', 'abc');
const foldQuery = query.fold('attribute', 'head');
const selectQuery = query.select('attribute1', 'attribute2');

Common query functions

These functions are available always, regardless of folding or selecting.

limit(number)
Sets limit of rows in response.
Request mutation: request.limit = count
Example
```
const limitedQuery = query.limit(20);
```
offset(number)
Sets how much rows to skip for response.
Request mutation: request.offset = count
Example
```
const offsetQuery = query.offset(20);
```
template(string)
Sets template in request. May determine which attributes are returned.
Request mutation: request.template = template
Example
```
const templatedQuery = query.template('workflow');
```

filter(string, FilterOperator, CoronerValueType)

Adds a filter to request.

Request mutation: request.filter[attribute] += [operator, value]

Example

const filteredQuery = query
    .filter('a', 'equal', 'xyz')
    .filter('timestamp', 'at-least', 1660000000)
    .filter('timestamp', 'at-most', 1660747935);

filter(string, QueryAttributeFilter[])

Adds filters to request. You can use this requests with Filters helper. See Fluent filters for more info.

Request mutation: request.filter[attribute] += filters

Example

// filter by timestamp
const filteredQuery = query.filter('timestamp', [
    ['at-least', 1660000000],
    ['at-most', 1660747935],
]);

// filter with Filters
const filteredQuery = query.filter('timestamp', Filters.time.from.last.hours(2).to.now());

table(string)
Sets the table name.
Request mutation: request.table = table
Example
```
// use 'metrics' table
query.table('metrics');
```
json()
Returns the build request. The request can be posted to any Coroner /api/query endpoint.
post(Partial<QuerySource>?)
Makes a POST call to Coroner with the built request.
Example
```
const response = await query.post();
if (!response.error) {
    // use the response
}
```

Select query functions

These functions are only available when selecting, and when the query is neither selected or folded.

select(...string[])
Adds provided attributes to the select request. You can add multiple attributes at once, or chain select calls.
Request mutation: request.select += [...attributes]
Example
```
const selectedQuery = query.select('a').select('b').select('c');
```
```
const selectedQuery = query.select('a', 'b', 'c');
```
select()
Returns the query as dynamic select. Use this to assign select attributes in runtime, without knowing the types.
Example
```
let query = query.select();
query = query.select('a');
query = query.select('b');
```
selectAll()
Selects all indexed attributes in table.
Request mutation: request.select_wildcard = { physical: true, virtual: true, derived: true }
Example
```
let query = query.selectAll();
```
selectAll(options)
Selects all available indexed attributes, physical, virtual, derived, or a combination of these.
Request mutation: request.select_wildcard = options
Example
```
let query = query.selectAll({ physical: true, virtual: false });
```

order(attribute, direction)

Adds order on attribute with direction specified.

Example

// This will order descending on attribute 'a', then ascending on attribute 'b'
query.select('a').select('b').order('a', 'descending').order('b', 'ascending');

Fold query functions

These functions are only available when folding, and when the query is neither selected or folded.

fold(string, ...FoldOperator)
Adds provided fold to the request.
Request mutation: request.fold[attribute] += [...fold]
Example
```
query.fold('fingerprint', 'head').fold('fingerprint', 'tail').fold('timestamp', 'distribution', 3);
```
fold()
Returns the query as dynamic fold. Use this to assign folds in runtime, without knowing the types.
Example
```
let query = query.fold();
query = query.fold('fingerprint', 'head');
query = query.fold('timestamp', 'tail');
```

removeFold(attribute, ...Partial<FoldOperator>)

Removes all previosuly added matching folds from the request.

Request mutation: request.fold[attribute] -= [...fold]

Example

const queryWithDistributions = query.fold('fingerprint', 'distribution', 3).fold('fingerprint', 'distribution', 4);

const queryWithoutDistributions = queryWithDistributions.removeFold('fingerprint', 'distribution');

group(attribute)
Sets the request group-by attribute. The attribute grouped by will be visible as groupKey in simple response.
Request mutation: request.group = [attribute]
Example
```
const groupedByFingerprint = query.group('fingerprint');
```

order(attribute, direction, ...FoldOperator)

Adds order on attribute fold with direction specified.

Example

// This will order descending on attribute 'a', fold 'head', then ascending on attribute 'a', fold 'tail'
query.fold('a', 'head').fold('a', 'tail').order('a', 'descending', 'head').order('a', 'ascending', 'tail');

orderByCount(direction)

Adds order on count with direction specified.

Example

// This will order descending on count
query.fold('a', 'head').fold('a', 'tail').orderByCount('descending');

orderByGroup(direction)

Adds order on group with direction specified.

Example

query.fold('a', 'head').fold('a', 'tail').groupBy('fingerprint').orderByGroup('descending');

having(attribute, index, operator, value)

Adds a post-aggregation filter on params specified.

Example

// Filters on 'head' fold
query.fold('a', 'head').having('a', ['head'], 'less-than', 123);

// Filters on 'distribution, 3' fold
query.fold('a', 'distribution', 3).having('a', ['distribution', 3], 'less-than', { keys: 123 });

having(attribute, index, operator, valueIndex, value)

Adds a post-aggregation filter on params specified.

Example

// Filters on 'head' fold
query.fold('a', 'head').having('a', 0, 'less-than', 0, 123);

// Filters on 'range' fold, value 'from'
query.fold('a', 'range').having('a', 0, 'less-than', 0, 123);

// Filters on 'range' fold, value 'to'
query.fold('a', 'range').having('a', 0, 'less-than', 1, 123);

having(attribute, fold, operator, valueIndex, value)

Adds a post-aggregation filter on params specified.

Example

// Filters on 'head' fold
query.fold('a', 'head').having('a', ['head'], 'less-than', 0, 123);

// Filters on 'range' fold, value 'from'
query.fold('a', 'range').having('a', ['range'], 'less-than', 0, 123);

// Filters on 'range' fold, value 'to'
query.fold('a', 'range').having('a', ['range'], 'less-than', 1, 123);

havingCount(operator, value)
Adds a post-aggregation filter on count.
Example
```
query.havingCount('greater-than', 10);
```
virtualColumn(name, type, params)
Adds a virtual column. The virtual column behaves like any other column.
Request mutation: request.virtual_columns += { name, type, [type]: params }
Example
```
// Adds a virtual column with name 'a'
query.virtualColumn('a', 'quantized_uint', { backing_column: 'timestamp', size: 3600, offset: 86400 });
```

Getting the request

At any point of querying, you can retrieve the raw request that is being built by using json function.

You can use this request to perform a query to Coroner.

const request = query.filter('a', 'equal', 'xyz').limit(20).select('b', 'c').json();

Getting the response

To get the response, you must select or fold at least once. After that, you can use the async post function, to receive the raw response from Coroner.

Check for error before trying to access the actual response.

const coronerResponse = await query.post();
if (!coronerResponse.success) {
    // An error happened!
    const message = coronerResponse.json().error.message;
    const code = coronerResponse.json().error.code;
    return;
}

// We got the raw response from Coroner query here!
const response = coronerResponse.json();

You can provide the source of data for making this query:

const coronerResponse = await query.post({
    address: 'http://sample.sp.backtrace.io',
    token: '00112233445566778899AABBCCDDEEFF',
    project: 'coroner',
});

Simple responses

To simplify reading the data from the response, there are two methods in every successful response: toArray and first. These will return a simplified key-value data structure representing the data received from Coroner.

The responses will be different for selecting and folding, respectively.

Simple select response

const coronerResponse = await query.select('a', 'b', 'c').post();
if (!coronerResponse.success) {
    return;
}

const firstRow = coronerResponse.first();
const rows = coronerResponse.all().rows; // this will contain the firstRow above as the first element
for (const row of rows) {
    console.log(row.a, row.b, row.c); // prints values of attributes a, b, c
}

Simple fold response

const coronerResponse = await request
    .fold('a', 'head')
    .fold('a', 'range')
    .fold('b', 'min')
    .fold('b', 'distribution', 3)
    .fold('b', 'distribution', 5)
    .group('c')
    .fold('c', 'bin', 3)
    .fold('c', 'unique')
    .post();

if (!coronerResponse.success) {
    return;
}

const firstRow = coronerResponse.first();
const rows = coronerResponse.all().rows; // this will contain the firstRow above as the first element
for (const row of rows) {
    console.log(row.count); // displays the group count
    console.log(row.attributes.c.groupKey); // displays the group key

    console.log(row.attributes.a.head[0].value, row.attributes.b.min[0].value); // displays the head and min values of attributes a and b
    console.log(row.attributes.a.range[0].value.from, row.attributes.a.range[0].value.to); // displays the range of attribute a

    console.log(row.attributes.c.unique[0].value); // displays the count of unique values of attribute c

    // displays all the details of [distribution, 3] of b attribute
    console.log(row.attributes.b.distribution[0].fold); // prints "distribution"
    console.log(row.attributes.b.distribution[0].rawFold); // prints ["distribution", 3]
    const distributionOfB3 = row.attributes.b.distribution[0].value;
    console.log(distributionOfB3.keys, distributionOfB3.tail);
    for (const value of distributionOfB3.values) {
        console.log(value);
    }

    // displays all the details of [distribution, 5] of b attribute
    const distributionOfB5 = row.attributes.b.distribution[1];
    console.log(distributionOfB5.value.keys, distributionOfB5.value.tail);
    for (const value of distributionOfB5.value.values) {
        console.log(value);
    }

    // displays all the details of bins of c attribute
    const binOfC = row.attributes.c.bin[0].value;
    for (const bin of binOfC.values) {
        console.log(bin.from, bin.to, bin.count);
    }
}

Available filters

These are all available filter operators that you can use in filter function.

String filters

at-least
at-most
contains
not-contains
equal
not-equal
greater-than
less-than
regular-expression
inverse-regular-expression
is-set
is-not-set

Number filters

at-least
at-most
equal
not-equal
greater-than
less-than
is-set
is-not-set

Boolean filters

equal
not-equal
is-set
is-not-set

Fluent filters

To simplify some attributes filtering, you can use Filters helper to create more complex filters with ease.

Time filters

Use Filters.time to create time filters with ranges.

import { Filters } from '@backtrace/forensics';

// Will return data from the last 2 hours
query.filter('timestamp', Filters.time.from.last.hours(2).to.now());

// Will return data from `fromDateObject` to `toDateObject`
query.filter('timestamp', Filters.time.from.date(fromDateObject).to.date(toDateObject));

Range filters

Use Filters.range to create filters with range.

// Will return values with _tx from range 100-500
query.filter('_tx', Filters.range(100, 500));

Ticket filters

Use Filters.ticket to create filters for tickets.

query.filter(Filters.ticket.state.isResolved());

By default it uses fingerprint;issues;state attribute to check status. If you are querying the issues table directly, use attribute():

query.table('issues').filter(Filters.ticket.attribute('state').isResolved());

Available folds

These are all available fold operators that you can use in the fold function.

Common folds

head - returns the first value in group,
tail - returns the last value in group,
unique - returns the number of unique values in group,
range - returns the range of values, from min to max,
max - returns the max value,
min - returns the min value,
distribution - returns the distribution of values, in number of specified buckets,
histogram - returns count of each discrete value,
object - returns highest row ID in the group.

String folds

String folds use only the common folds.

Number and boolean folds

In addition to common folds, there are additional folds available:

mean - returns the mean value of all values in group,
sum - returns the sum of all values in group,
bin - ? not sure how to describe this, help.

Available `having` filters

These are all available having operators that you can use in the having function.

== (equal)
!= (not equal)
< (less-than)
> (greater-than)
<= (at-least)
>= (at-most)

Filters in parentheses are supported only by Forensics, and not by Coroner itself.

Available virtual column types

These are all available virtual column ttypes that you can use in the virtualColumn function.

quantize_uint

Options:

{
    backing_column: string;
    size: number;
    offset: number;
}

truncate_timestamp

Options:

{
    backing_column: string;
    granularity: 'day' | 'month' | 'quarter' | 'year';
}

Overriding the API caller

By default, the query maker is using http/https modules in Node, and XMLHttpRequest in browser.

If you want to use your own implementation for making the query, provide an implementation of ICoronerApiCallerFactory to the Forensics options. The factory should create your ICoronerApiCaller

A Typescript class implementation of an API caller using axios may look like this:

import axios from 'axios';
import { ICoronerQueryMaker } from '@backtrace/forensics';

class AxiosCoronerQueryMaker implements ICoronerApiCaller {
    public async post<R extends QueryResponse>(
        url: string | URL,
        body?: string,
        headers?: Record<string, string>,
    ): Promise<R> {
        const response = await axios.post<R>(url, body, { headers });
        return response.data;
    }
}

Plugins

As of version 0.6.0, @backtrace/forensics supports writing plugins. Plugins pack multiple extensions that can later extend a Forensics instance.

Importing a plugin

Pass the plugin into options while creating the instance:

const instance = new Forensics({
    plugins: [myPlugin()],
});

const response = await instance
    .addHeadAndTailsFold('attribute')
    .addHourlyQuantizedColumn('timestamp', 'quantized_timestamp')
    .group('quantized_timestamp')
    .post();

Writing a plugin

Import Plugins namespace from @backtrace/forensics, and use the Plugins.createPlugin function to begin. Add each extension as a element to the plugin, for example:

export function myPlugin() {
    return Plugins.createPlugin(
        Plugins.extendFoldCoronerQuery((context) => ({
            addHeadAndTailFolds(attribute: string) {
                return this.fold(attribute, 'head').fold(attribute, 'tail');
            },
        })),
        Plugins.extendFoldedCoronerQuery((context) => ({
            addHourlyQuantizedColumn(attribute: string, name: string) {
                return this.virtualColumn(name, 'quantized_uint', {
                    backing_column: attribute,
                    size: 3600,
                    offset: 86400,
                });
            },
        })),
    );
}

Adding extensions

Use methods in the Plugins namespace and pass them to createPlugin:

addQueryExtension
addFoldQueryExtension
addFoldedQueryExtension
addSelectQueryExtension
addSelectedQueryExtension
addResponseExtension
addFailedResponseExtension
addSuccessfulResponseExtension
addFoldResponseExtension
addFailedFoldResponseExtension
addSuccessfulFoldResponseExtension
addSelectResponseExtension
addFailedSelectResponseExtension
addSuccessfulSelectResponseExtension

Each function will add functions to a specific query or response interface.

Contact

Code: Sebastian Alex

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

backtrace-forensic

Building

Running tests

Usage

Creating a query

Options

Using the query functions

Common query functions

Select query functions

Fold query functions

Getting the request

Getting the response

Simple responses

Simple select response

Simple fold response

Available filters

String filters

Number filters

Boolean filters

Fluent filters

Time filters

Range filters

Ticket filters

Available folds

Common folds

String folds

Number and boolean folds

Available having filters

Available virtual column types

Overriding the API caller

Plugins

Importing a plugin

Writing a plugin

Adding extensions

Contact

Available `having` filters