text2json-parser

A customizable text to json parser utility, a very simple approach.

This is still a trial I am using for a private project. The idea was to be able to parse different text files to JSON format without coding a solution ad-hoc for each file. To do so it is necessary to describe a JSON file to describe the operations we want to process on the raw data.

This function expects two parameters (described below) and the output will be a JSON with the file identifier as key and the JSON entries as values.

Files

A JSON file that allocates the files to process. The dictionary keys are the identifiers/names/paths of the files and the values of the file content as a string or buffer.

<pre>
let filesJson = {
    logs.txt: '<FILE-CONTENT-BUFFER>',
    operations.log: '<FILE-CONTENT-BUFFER>'
}
</pre>

JSON files descriptor

This file describes all the operations we want to make on raw files to parse the contents. Here is the description of each field.

• auxiliarFields: These are fields that we might use in the parsing process.
• commonFields: These are fields that will be added to all the entries we are gonna parse.
• files: The files we will parse with name as key and the entries list as values.

File descriptor

Fields

• name: The name of the field
• source: The source of the field (depending on the source type we would need diferent parameters).

|Source|Description|Required fields| |---|---|---| |FILE|From a file content|filename| |FILENAME|From a file name|filename| |LITERAL|From a literal value set by the user|value| |ROW|From a splitted file row| |

Procedures

To parse a field we can add three types of procedures

|Procedure|Description|Value fields| |---|---|---| |extract|Initial extraction operation|operation description| |process|Additional operations to perform on the previous extracted field value|operation descriptions list| |conditional|Conditions needed to add the field to the output entries|conditional descriptor|

Operations

|Operation|Description|Additional fields|expectedValues| |---|---|---|---| |EXTRACT|Extract value using a regular expression|regex(string)| |SUBSTRING|Substring operation on a string parameter|from, to (string:value/number:index)| |PARSE|Parse value to another field type|from, to| 'STRING', 'NUMBER', 'DATE', 'LIST'| |REPLACE|Parse value to another field type|from, to|| |SPLIT|Parse value to another field type|by|| |JOIN|Parse value to another field type|values, by|values:[], by: (joiner) |

Conditionals

|Conditionals|Additional fields|expectedValues| |---|---|---| |EQUALS|regex(string)| |NOT_EQUALS|from, to (string:value/number:index)| |LIKE|from, to| 'STRING', 'NUMBER', 'DATE', 'LIST'| |NOT_LIKE|Parse value to another field type|from, to|| |IN|Parse value to another field type|by|| |NOT_IN|Parse value to another field type|values, by|values:[], by: (joiner) | |STARTSWITH|Parse value to another field type|by|| |ENDSWITH|Parse value to another field type|values, by|values:[], by: (joiner) |

Field descriptor

Fields are used to describe the parameters we will extract from our sources.

<pre>
const filesDescriptorJson = {
  auxiliarFields: [
    {
      name: 'year', source: 'FILE', filename: 'journalctl.log',
      extract: { operation: 'EXTRACT', regex: /\d{4}/ }
    }
  ],
  commonFields: [
    {
      name: 'session', type: 'STRING', source: 'FILENAME', filename: 'rosout.log',
      extract: { operation: 'SUBSTRING', from: undefined, to: '/' }
    },
    {
      name: 'serial', type: 'STRING', source: 'FILE', filename: 'journalctl.log',
      extract: { operation: 'EXTRACT', regex: /\w{2}-\w{4}-\w{3}-\w{6}/ }
    }
  ],
  files: {
    rosout: {
      filename: 'rosout.log', lineDelimiter: /(\d{10}\.\d{9})/,
      fields: [
        { name: 'file', source: 'LITERAL', value: 'rosout.log' },
        { name: 'raw', type: 'STRING', source: 'ROW' },
        {
          name: 'time', type: 'DATE', source: 'ROW',
          extract: { operation: 'SUBSTRING', from: 0, to: ' ' },
          process: [
            { operation: 'REPLACE', from: '.', to: '' },
            { operation: 'SUBSTRING', from: 0, to: 13 },
            { operation: 'PARSE', from: 'NUMBER', to: 'DATE' },
          ]
        },
        {
          name: 'severity', type: 'STRING', source: 'ROW',
          conditions: [{ key: '$this', operator: 'IN', value: ['INFO', 'WARN', 'ERROR', 'FATAL'] }],
          extract: { operation: 'SUBSTRING', from: 0, to: ' ' }
        },
        {
          name: 'node', type: 'STRING', source: 'ROW',
          conditions: [{ key: '$this', operator: 'STARTSWITH', value: '/' }],
          extract: { operation: 'SUBSTRING', from: 0, to: ' ' }
        },
        {
          name: 'source', type: 'STRING', source: 'ROW',
          conditions: [
            { key: '$this', operator: 'STARTSWITH', value: '[' },
            { key: '$this', operator: 'ENDSWITH', value: ')' }
          ],
          extract: { operation: 'SUBSTRING', from: 0, to: ' ' }
        },
        {
          name: 'topics', type: 'LIST', source: 'ROW',
          extract: { operation: 'SUBSTRING', from: '[topics:', to: ']' },
          conditions: [{ key: '$this', operator: 'STARTSWITH', value: '/' }],
          process: [{ operation: 'SPLIT', by: ', ' }]
        },
        { name: 'message', type: 'STRING', source: 'ROW' },
        {
          name: 'mongoTime', type: 'DATE', source: 'ROW',
          conditions: [{ key: '$node', operator: 'EQUALS', value: '/mongo_server' }],
          extract: { operation: 'SUBSTRING', from: 0, to: ' ' },
          process: [{ operation: 'PARSE', from: 'STRING', to: 'DATE' }]
        },
        {
          name: 'mongoProcess', type: 'DATE', source: 'ROW',
          conditions: [{ key: '$node', operator: 'EQUALS', value: '/mongo_server' }],
          extract: { operation: 'SUBSTRING', from: '[', to: ']' }
        },
        {
          name: 'mongoMessage', type: 'DATE', source: 'ROW',
          conditions: [{ key: '$node', operator: 'EQUALS', value: '/mongo_server' }]
        },
      ]
    },
    </pre>