API Indexer

This library will analyze the blueprints and construct a a node path matching system that allows the attributes in a data update to be formatted for the ElasticSearch index update.

Install

npm install --save @leverege/api-indexer

Usage

Before the indexer can be used, the Indexer.groovy script must be uploaded to Elastic Search.

const scriptId = 'indexer'
await ScriptFile.uploadIndexerScripts( client, scriptId )

Next, create a Blueprints store to manage the Blueprint objects. The source is used to retrieve the blueprint data and normally invokes Api library to get them.

const source = {
  get( projectId, id ) {
    return api.project( projectId ).blueprint( id ).get()
  }
}

const blueprints = new Blueprints( { source } )

Once you have this, retrieve the Blueprint from it when you need to process a deviceDataEventMsg. Invoke the BlueprintIndexer to get back an IndexRequest. This request can then be given to IndexUpdate to run.

// This is an efficient way to get the blueprint if it is ready for use without having to await.
const bp = blueprints.getIfReady( blueprintId ) || await blueprints.get( projectid, blueprintId )

// Invoke the indexer and create the Bulk commands
const opts = { indexUpdated : true, writeDataMetadata : false }
const idxReq = BlueprintIndexer.createIndex( pathValueArray, time, bp, opts ) // time could also be { time, source : { ... } }
const bulk = IndexUpdate.createBulkCommands( { id : deviceId, index : deviceIndex, idxReq, scriptId } )

// Invoke ElasticSearch to index it
const br = await ES.bulk( { refresh : true, body : bulk } )

Simple Indexing

Attributes in the blueprint will are indexed will be placed into the idx field on the document. For non-collection attributes (attributes whose paths contain no * or @ symbols), there is a simple mapping between these fields.

| Attribute Type | Extension | |----------------|-----------| | boolean | -b | | int | -n | | number | -n | | percent | -n | | <unit> | -n | | string | -s | | enum | -k | | resourceEnum | -k | | date | -d | | timestamp | -d | | geoPoint | -gp | | geoShape | -gs | | geoJson | -gs | | point | -cp | | shape | -cs | | version | -v | | ip | -ip |

A few other fields are indexed as well, with these extension

| Field | Extension | |-------|-----------| | UUID reference such as parentIds, blueprintIds, etc | -k | | keyword | -k | | Time a field was updated | -u |

When a request to set blueprint attributes occurs, the result indexed fields will look something like this:

// Path / Value request
const msg = [
  { path : 'name', value : 'Stephen' },
  { path : 'favoriteNumber', value : 12 },
  { path : 'geo/position', value : { lat : 34.0, lon : -78.0 } }
]

const idx = blueprintIndexer.createIndex( msg, time, blueprint )
expect( idx.getIndex() ).to.deep.equal( {
  'name-s' : 'Stephen',
  'name-u' : time,
  'favoriteNumber-n' : 12,
  'favoriteNumber-u' : time,
  geo : {
    'position-gp' : { lat : 34.0, lon : -78.0 },
    'position-u' : time
  }
} )

The -u extensions represent the time when the field was set.

Attribute Primitive Arrays

An attribute with the path name values* indicate that multiple values can be stored in an array at the field name values. Because ElasticSearch supports arrays natively, setting of the values* attribute in its entirety results in a simple set, as descripted above. The extension ( -n, -s, etc ) is based on the attribute's type according the table above. For example:

const msg = [ { path : 'values', value : [ 14, 17, 20 ] } ]

const idx = blueprintIndexer.createIndex( msg, time, blueprint )

expect( idx.getIndex() ).to.deep.equal( {
  'values-n' : [ 14, 17, 20 ],
  'values-u' : time
} )

Although not recommended, You can update a single element in this array. If you need to update only one field in the array and you do not know thw values of the other elements in the array, you an use path with an index in it:

const msg = [ { path : 'values/1', value : 18 } ]

This will result in a scripting action instead of a simple ElasticSearch index option. See Scripting definitions below for a description. When this is called, the array must be able to hold the index specified. If an array index of 10 is given, elements zero through nine will be set to null if they do not already exist.

Attribute Primitive Maps

An attribute with the path name ending in @ (such as values@) indicate that multiple values can be stored in a map at the field name values. In order to support this, the extension type for the root is appended with -attrmap, and a Nested object array is used. Each element in the array contain a key, value-<ext> and value-u. If the attribute is set in its entirety, this can be inserted using the simple, non-scripted mechanism. For example:

const msg = [ { path : 'values', value : { battery1 : 100, battery2 : 97.7 } } ]

const idx = blueprintIndexer.createIndex( msg, time, blueprint )

expect( idx.getIndex() ).to.deep.equal( {
  'values-attrmap' : [ 
    { key : 'battery1', 'value-n' : 100, 'value-u' : time }
    { key : 'battery2', 'value-n' : 97.7, 'value-u' : time }
  ],
  'values-u' : time
} )

Like the array, the Map can also be updated individually.

const msg = [ { path : 'values/battery2', value : 81.2 } ]

Setting a key to null will remove the entry completely from the attribute map list.

Map Collections

Map collections are paths whose non-leaf path contain the @ at the end of a path name. For example, a set of attributes such as:

  { name : 'Module Battery', path : 'module@/battery', type : 'percent' },
  { name : 'Module Temp', path : 'module@/temperature', type : 'temperature' }

indicates that ther can be many modules, each with a name (the key) and battery and temperature. The data for this might look like:

{
  module : {
    engine : { battery : 92, temperature : 70 },
    fan : { battery : 45, temperature : 50 },
  }
}

In the idx field in elastic search, this is store in a nested array and would look like:

{
  idx : {
    module-map : [
      { key : 'engine', 'battery-n' : 92, 'battery-u' : <time>, 'temperature-n' : 70, 'temperature-u' : <time> },
      { key : 'fan', 'battery-n' : 45, 'battery-u' : <time>, 'temperature-n' : 50, 'temperature-u' : <time> }
    ]
  }
}

The map can be updated in its entirety, or at one key in its entirety, or at individual fields in the map. The path values would look like:

// The entire map
{ 
  path : 'module', 
  value : {
    engine : { battery : 92, temperature : 70 },
    fan : { battery : 45, temperature : 50 },
  }
}

// One key in map
{ 
  path : 'module/fan', 
  value : { battery : 40, temperature : 49 },
}

// Individual field
{ path : 'module/engine/battery', value : 91 }

To remove the key from the map, a path of moduel/<key> with a value of null must be sent. This will completely free up the indexing space for that key. Setting all attributes in the key to null will still retain the field updated timestamps for each attribute.

Array Collections

Array collections are paths whose non-leaf path contain the * at the end of a path name. For example, a set of attributes such as:

  { name : 'Array Battery', path : 'fields*/battery', type : 'percent' },
  { name : 'Array Temp', path : 'fields*/temperature', type : 'temperature' }

indicates that ther can be many modules, each with a battery and temperature. The data for this might look like:

{
  fields : [
    { battery : 92, temperature : 70 },
    { battery : 45, temperature : 50 },
  ]
}

In the idx field in elastic search, this is store in a nested array and would look like:

{
  idx : {
    fields-arr : [
      { index : 0, 'battery-n' : 92, 'battery-u' : <time>, 'temperature-n' : 70, 'temperature-u' : <time> },
      { index : 1, 'battery-n' : 45, 'battery-u' : <time>, 'temperature-n' : 50, 'temperature-u' : <time> }
    ]
  }
}

Like the map, the array can be updated in its entirety, or at one key in its entirety, or at individual fields in the map. The path values would look like:

// The entire map
{ 
  path : 'fields', 
  value : [
    { battery : 92, temperature : 70 },
    { battery : 45, temperature : 50 },
  ]
}

// One key in map
{ 
  path : 'fields/0', 
  value : { battery : 40, temperature : 49 },
}

// Individual field
{ path : 'fields/0/battery', value : 91 }

Unlike the map however, setting a value to null will insert a null into the array instead of removing it and shifting indices. If you wish to accomplish the 'shift' behavior, you will need to set the entire array yourself.

{ path : 'fields/0/battery', value : null }

will result in

{ fields : [ null, { battery : 45, temperature : 50 }, ] }

Scripts

EnsureGroup

• create the HashMap if not already there
• Holds commands against the group

EnsureArray

• create the ArrayList if not already there
• Holds commands against the group

EnsureMap

• create the Array if not already there
• Holds commands against the group

EnsureKey

• Makes sure the Map's key exists
• Holds commands against the key

EnsureIndex

• Makes sre the Array's key exists
• Holds commands against the entry

SetValue

• Sets the key to the given value

Data representations and undefined attributes

The data object stored in the Elastic Search document is meant to closely match the form of the data. In particular, Maps are represented as JSON object, and the type extensions are removed. Because the path/value object methods allow setting of fields at various levels in the object hierarchy, the data object might need to be scripted as well. Further, the data might contain data that is not defined in the blueprint itself. There are a few special situations that arise from this that need to be addressed.

Map vs Array

Consider the follow path/value objects. (Please note, these in the )

{ path : 'value', value : [ 4, 5, 6 ] }

{ path : 'value/0', value : 1 },

{ path : 'value/bar', value : 2 }

The way this would be handled is that value is initially an array, the array is then updated, and then on the last set, the value with key bar to 2. For data that does not have attributes defined, this behavior will follow Firebase's behavior, which will be to promote the array to a map. The indices will be used as keys in the new map.

For attribute paths that are defined as arrays, the array will not be promoted and Elastic Search will reject the change.