token-types

A primitive token library used to read and write from a node Buffer. Although it is possible to use this module directly, it is primary designed to be used with strtok3 tokenizer.

Compatibility

Module: version 5 migrated from CommonJS to pure ECMAScript Module (ESM). JavaScript is compliant with ECMAScript 2019 (ES10).

Installation

npm install --save token-types

Usually in combination with strtok3:

npm install --save strtok3

Using TypeScript you should also install @tokenizer/token as a development dependency:

npm install --save-dev @tokenizer/token

Example

import * as strtok3 from 'strtok3';
import * as token from 'token-types';
    
(async () => {

  const tokenizer = await strtok3.fromFile("somefile.bin");
  try {
    const myNumber = await tokenizer.readToken(token.Float32_BE);
    console.log(`My number: ${myNumber}`);
  } finally {
    tokenizer.close(); // Close the file
  } 
})();

Tokens

Numeric tokens

node-strtok supports a wide variety of numerical tokens out of the box:

| Token | Number | Bits | Endianness | |---------------|------------------|------|----------------| | UINT8 | Unsigned integer | 8 | n/a | | UINT16_BE | Unsigned integer | 16 | big endian | | UINT16_LE | Unsigned integer | 16 | little endian | | UINT24_BE | Unsigned integer | 24 | big endian | | UINT24_LE | Unsigned integer | 24 | little endian | | UINT32_BE | Unsigned integer | 32 | big endian | | UINT32_LE | Unsigned integer | 32 | little endian | | UINT64_BE | Unsigned integer | 64 | big endian | | UINT64_LE* | Unsigned integer | 64 | little endian | | INT8 | Signed integer | 8 | n/a | | INT16_BE | Signed integer | 16 | big endian | | INT16_LE | Signed integer | 16 | little endian | | INT24_BE | Signed integer | 24 | big endian | | INT24_LE | Signed integer | 24 | little endian | | INT32_BE | Signed integer | 32 | big endian | | INT32_LE | Signed integer | 32 | little endian | | INT64_BE | Signed integer | 64 | big endian | | INT64_LE* | Signed integer | 64 | little endian | | Float16_BE | IEEE 754 float | 16 | big endian | | Float16_LE | IEEE 754 float | 16 | little endian | | Float32_BE | IEEE 754 float | 32 | big endian | | Float32_LE | IEEE 754 float | 32 | little endian | | Float64_BE | IEEE 754 float | 64 | big endian | | Float64_LE | IEEE 754 float | 64 | little endian | | Float80_BE* | IEEE 754 float | 80 | big endian | | Float80_LE* | IEEE 754 float | 80 | little endian |

Other tokens

String types:

• Windows-1252
• ISO-8859-1

*) The tokens exceed the JavaScript IEEE 754 64-bit Floating Point precision, decoding and encoding is best effort based.

Custom token

Complex tokens can be added, which makes very suitable for reading binary files or network messages:

 ExtendedHeader = {
    len: 10,

    get: (buf, off) => {
      return {
        // Extended header size
        size: Token.UINT32_BE.get(buf, off),
        // Extended Flags
        extendedFlags: Token.UINT16_BE.get(buf, off + 4),
        // Size of padding
        sizeOfPadding: Token.UINT32_BE.get(buf, off + 6),
        // CRC data present
        crcDataPresent: common.strtokBITSET.get(buf, off + 4, 31)
      };
    }
  };