Declarative Mapper

Overview

Declarative Mapper is a JSON data transformation and object mapping library for JavaScript/TypeScript. Define declarative mapping templates to convert documents quickly, with schema helpers and safe VM-based execution.

Table of Contents

• Overview
  • Reasoning
  • Quick Start Example
  • Compatibility
• Mapping Instructions
  • Runtime Variables Quick Reference
  • Objects
  • Arrays
  • String[] from Object[]
  • String[] from String[]
  • Tuple Arrays
  • Context Switching
  • Dynamic Output Keys
• Extensions
• Complex Mapping Example

Reasoning

On several projects, I needed a library that could convert one JSON format to another (for example, an invoice from one system into another). It had to support declarative mapping instructions so users could configure mappings from a UI. It also had to be flexible enough for complex requirements, secure against JS injection, and fast enough to process streams with millions of records.

That is where Declarative Mapper came in:

• Declarative - declarative mapping instructions allow configuration from a UI. In simple scenarios, no technical knowledge is needed.
• Flexible - runs JavaScript under the hood to support complex instructions.
• Secure - restricts access to the outside environment by executing mappings in a separate V8 Virtual Machine context.
• Fast - mapping instructions are compiled once up front, allowing processing at ~200k objects/sec on Apple M1 Pro.
• Typed - written in TypeScript
• Lightweight - no dependencies

Quick Start Example

import { createMapper } from 'declarative-mapper';

// Source records from system A
const sourceOrders = [
  {
    orderId: 'SO-1001',
    customerName: 'Acme Ltd',
    lineItems: [{ sku: 'A-1', qty: 2, unitPrice: 10 }]
  },
  {
    orderId: 'SO-1002',
    customerName: 'Globex',
    lineItems: [{ sku: 'B-9', qty: 1, unitPrice: 25 }]
  }
];

// Compile once
const mapper = createMapper({
  id: 'orderId',
  customer: 'customerName',
  items: {
    forEach: 'lineItems',
    map: {
      code: 'sku',
      quantity: 'qty',
      amount: 'qty * unitPrice'
    }
  },
  totalAmount: 'lineItems.reduce((sum, i) => sum + (i.qty * i.unitPrice), 0)'
});

// Use in a loop; 200k+ objects/sec
const results = sourceOrders.map(mapper); 

Compatibility

• Node.js: 16+
• Browser: best effort support; requires a vm polyfill

Mapping Instructions

In mapping JSON, the left side is a key in the resulting object. The right side is either a string with a valid JS expression or an object with mapping instructions.

{
  "key": "100",               // numeric value, produces `"key": 100`
  "key": "true",              // boolean value, produces `"key": true`
  "key": "'text'",            // text value, produces `"key": "text"` (notice inner quotation marks)
  "key": "foo",               // access to an input variable `foo`
  "key": "Number(foo)",       // access to an input variable `foo` converted to a number, produces `"key": 100`
  "key": "arr.map(e => ...)", // more complex JS expression that produces an array
  "key": {                    // object mapping, produces `{ foo: 'bar' }`
    "foo": "'bar'"
  }, 
  "key": {                    // same as above, but more verbose
    "map": { /*...*/ }
  },
  "key": {                    // array mapped from input
    "forEach": "nested.array.filter(e => !!e)",
    "map": { /*...*/ }        // $record, $index, $collection are available in "map"
  },
  "key": {                    // tuple array
    "0": { /*...*/ },
    "1": { /*...*/ },
    "5": { /*...*/ }
  },
  "key": {                    // object mapping from a different context 
    "from": "some.nested.field",
    "map": { /*...*/ }
  },
  "${prefix}_${id}": "value", // dynamic output key (template interpolation)
}

Runtime Variables Quick Reference

| Variable | Description | Available in | | --- | --- | --- | | $input | Entire source document passed to the mapper. | All mapping contexts | | $record | Current element in a forEach iteration. | forEach → map | | $index | Current element index in a forEach iteration. | forEach → map | | $collection | Entire source array selected by forEach. | forEach → map |

Objects

Mapping of an object with inner properties:

  "key": {
    "foo": "-1"
  }

or

  "key": {
    "map": {
      "foo": "-1"
    }
  } 

Both examples above produce the same result (the second one is more verbose, but keeps a consistent format with array mappings):

  "key": {
    "foo": -1
  } 

Arrays

Assume we have input with an array of objects and need to produce one output object per element. In such cases, the "forEach": "", "map": {} construction can be used:

{
  "inputArray": [{
    "arrayInnerProp": "value1"
  }, {
    "arrayInnerProp": "value2"
  }]
}

  "key": {
    "forEach": "inputArray",
    "map": {
      "foo": "arrayInnerProperty"
    }
  }

Result:

  "key": [{
    "foo": "value1"
  }, {
    "foo": "value2"
  }]

In this mapping, the execution context shifts to each input object, so inner properties can be referenced directly as arrayInnerProperty instead of inputArray[index].arrayInnerProperty.

Inside forEach mappings, $record, $index, $collection, and $input are also available. More on that in Context Switching.

String[] from Object[]

If the array should contain plain values instead of objects, use "*" on the left side instead of a key name:

  "key": {
    "forEach": "inputArray",
    "map": {
      "*": "arrayInnerProperty"
    }
  }

Produces:

  "key": [
    "value1",
    "value2"
  ]

String[] from String[]

Arrays with simple values can be mapped the same way, and the current iterated element is available as $record:

{
  "inputValues": [1, 2, 3]
}

{
  "forEach": "inputValues",
  "map": {
    "*": "$record * 2"
  }
}

Result:

[2, 4, 6]

Tuple Arrays

If the array should have a predefined set of elements, each element can be mapped by index:

  "key": {
    "0": {
      "foo": "\"text1\""
    },
    "2": "1000"
  }

Result:

  "key": [
    {
      "foo": "text1"
    },
    null,
    1000
  ]

Context Switching

When arrays are mapped with the "forEach": "", "map": {} statement, the execution context automatically switches to the objects selected by forEach (see above). A similar technique is useful when many properties need to be mapped from an object outside the current context. In that case, use the "from": "", "map": {} statement.

Down in the source tree:

  "key": {
    "from": "field.innerArray[0].innerObject",
    "map": {
      "foo": "nestedProperty"
    }
  }

Or up in the source document:

  "key": {
    "from": "$input.rootLevelProperty",
    "map": {
      "foo": "nestedProperty"
    }
  }

Inside from mappings, you can still reference root-level fields through $input.

Runtime variables:

• $record - current element of the array being iterated with forEach
• $index - index of the current array element
• $collection - entire collection of the elements being iterated
• $input - entire document passed as mapping input

Combined example (forEach + root reference):

{
  "forEach": "LINE_ITEMS",
  "map": {
    "lineNo": "$index + 1",
    "sourceId": "$input.id",
    "raw": "$record"
  }
}

Dynamic Output Keys

You can build output property names dynamically with template-based keys on the left side. This works in regular mappings, forEach mappings, and from mappings.

{
  "${prefix}_${id}": "value"
}

For input:

{
  "prefix": "item",
  "id": 7,
  "value": "abc"
}

Output:

{
  "item_7": "abc"
}

To keep ${...} as a literal key (without interpolation), escape it with a leading backslash:

{
  "\\${prefix}_${id}": "value"
}

Extensions

Use extensions to pass helper functions and lookup data into mapping expressions:

const mapper = createMapper({
  code: 'catalog[itemId]',
  normalizedName: 'normalize(name)'
}, {
  extensions: {
    catalog: { A1: 'SKU-001' },
    normalize: (s: string) => s.trim().toUpperCase()
  }
});

Extension keys are available as globals in expressions.

If an extension key conflicts with an input field name, mapper throws an error.

Complex Mapping Example

// Some kind of a document we expect on input
const input = {
  LINE_ITEMS: [
    { UPC: '123', QTY: 1, PRICE: 3.4 },
    { UPC: '456', QTY: 2, PRICE: 5.7 }
  ],
  ALLOWANCES: [
    { ITEM_UPC: '123', AMOUNT: 1.5 }
  ]
};

// Additional information we want to pass to the mapping environment
const itemCatalog = [
  { upc: '123', vendorCode: 'X-123' },
  { upc: '456', vendorCode: 'X-456' }
];

// Some format we need
const desiredOutput = {
  title: 'Invoice 1',
  items: [
    {
      code: 'X-123',
      qty: 1,
      price: 3.4,
      amount: 3.4,
      allowances: [1.5]
    },
    {
      code: 'X-456',
      qty: 2,
      price: 5.7,
      amount: 11.4,
      allowances: []
    }
  ],
  total: 14.8
};


// Declarative instructions on how to convert the input format
// to the desired format
const mapping = {
  // mapping to a constant
  title: '"Invoice 1"',

  // array mapping from another array
  items: {
    forEach: 'LINE_ITEMS',
    map: {
      // data lookup from an additional source passed to `extensions`
      code: 'itemCatalog.find(e => e.upc === UPC).vendorCode',
      
      // fields mapping in a context of `LINE_ITEMS` elements
      qty: 'QTY',
      price: 'PRICE',
      amount: 'QTY * PRICE',

      // data mapping from a source different from the current mapping context
      // (ALLOWANCES are placed next to LINE_ITEMS in the input)
      allowances: {
        forEach: 'ALLOWANCES.filter(a => a.ITEM_UPC === UPC)',
        map: {
          '*': 'AMOUNT'
        }
      }
    }
  },

  // property mapping from an array,
  // with a custom reducer function defined in `extensions`
  total: '$sum(LINE_ITEMS, i => i.QTY * i.PRICE)'
};

// Pre-compiled function that can be executed any number of times
const mapper = createMapper(mapping, {
  extensions: {
    itemCatalog,
    $sum: (arr, cb) => arr.reduce((t, el) => t + cb(el), 0)
  }
});

const result = mapper(input);

expect(result).to.eql(desiredOutput);