json-xform

JSON transform 🤖

Overview

A library to transform a JSON structure to another one by using an intermediate JSON DSL. It shall facilitate cases where an application (or an API) needs to process several JSON files generated by other software clients and to transform it to an internal, common format for further processing.

An example for such a case would be an application which takes reports, perhaps from testing tools and generates tickets or tasks (in Jira for instance). In order to avoid coding each time when a new tool shall be integrated, the user can provide a mapping by implementing the appropriate DSL alongside with the JSON report to process.

The DSL

The DSL is implemented in JSON. Its vocabulary is limited to a small set of words.

1. fieldset - defines an array of objects, each object encapsulates
2. from - defines the field to get the value from the source JSON.
3. to - defines the field to write the value to the target JSON.
4. withTemplate - defines an arbitrary string with template placeholders which hold references to fields in the source object to get values from.
5. fromEach - defines an object which addresses an array in the source JSON and provides the possibility to pick particular source fields to write to the target JSON by using the fieldset again.
6. field - defines the field in the fromEach block to get the value from.
7. flatten - to flatten collections.
8. fromArray - defines an ordered list of source paths whose values are collected into the array at to. Counterpart to from for list outputs.

fromArray — assembling a list from multiple sources

from reads one source path into a scalar target. fromArray reads N source paths into a list target, in the order given. Useful whenever the result should be a positional array and the source data is scattered across distinct fields rather than already laid out as a collection.

const source = {
  card: { url: 'https://card.jpg' },
  event: { url: 'https://event.jpg' }
};

const mapping = {
  fieldset: [
    {
      fromArray: ['card.url', 'event.url'],
      to: 'imageUrls'
    }
  ]
};

// → { imageUrls: ['https://card.jpg', 'https://event.jpg'] }

Modifiers

All optional. Defaults are no-op so behaviour matches the rest of the DSL out of the box; turn each one on when you need it.

• skipEmpty: true — drops undefined, null, and empty-string elements from the result. Preserves 0, false, [], and {} so positional booleans and numerics are not lost.
• unique: true — deduplicates while preserving first-seen order. String / number / boolean dedup by value; objects dedup by reference (standard JS Set semantics).
• flatten: true — when an element resolves to an array, splice its values into the output instead of nesting. One level deep, matching the existing fromEach.flatten behaviour.
• via: { ... } — applied per element rather than to the array as a whole, so date / commands transforms work the way you'd expect when the source values are individually formatted (e.g. a list of dates).

The four modifiers compose in a fixed order: via runs first, then flatten, then skipEmpty, then unique. That order keeps the intuitive semantics — format each value, splice array results, drop the emptiness that the formatting may have produced, finally dedup.

When fromArray is required vs. recommended

fromArray always needs an explicit to. There is no implicit derivation rule for it the way single-path from derives the target from the source path when to is omitted.

If toArray: true appears on the same entry it is ignored — the output is already an array, so wrapping would just nest the result.

If two fieldset entries write to the same to, the first one wins (this is the existing addPropToTarget behaviour, not a fromArray special case). Use distinct to paths if you need both to land.

Dependencies

The essential libraries used by this project are jsonpath and jsonschema

How it works

There are 2 functions exposed which accept two JSON objects. One JSON object contains the mapping while the other one contains the source object.

mapToNewObject - Accepts JSON obects.

mapWithTemplate - Accepts JSON files.

Both functions return the transform JSON object.

Mapping

Let's say you have a JSON file which looks something like the following:

    const source = {
      highLevel: [
        {
          fieldOne: 1,
          fieldTwo: 2,
          thisDate: '1981-03-10',
          lowLevel: [
            {
              fieldThree: 3,
              fieldFour: 4,
              basement: [
                {
                  this: {
                    thing: {
                      there: 'here I am'
                    }
                  }
                }
              ]
            }
          ]
        }
      ]
    };

And for some reason (only you can know) you'd like to have something like this instead:

    const target = {
      flat: [
        {
          fieldOne: 1,
          fieldTwo: [2],
          newProp: '1 is not 2',
          thatDate: '10/03/1981',
          fieldThree: 3,
          fieldFour: 4,
          that: {
            here: 'here I am'
          }
        }
      ]
    };

You can do so by providing this mapping:

const mapping = {
      fieldset: [
        {
          fromEach: {
            field: 'highLevel',
            to: 'flat',
            flatten: true,
            fieldset: [
              {
                from: 'fieldOne'
              },
              {
                from: 'fieldTwo'
                toArray: true
              },
              {
                to: 'newProp',
                withTemplate: '${fieldOne} is not ${fieldTwo}'
              },
              {
                from: 'thisDate',
                to: 'otherDate',
                via: {
                    type: 'date',
                    sourceFormat: 'yyyy-MM-dd',
                    format: 'dd/MM/yyyy'
                }
              },
              {
                fromEach: {
                  field: 'lowLevel',
                  flatten: true,
                  fieldset: [
                    {
                      from: 'fieldThree'
                    },
                    {
                      from: 'fieldFour'
                    },
                    {
                      fromEach: {
                        field: 'basement',
                        fieldset: [
                          {
                            from: 'this.thing.there',
                            to: 'that.here'
                          }
                        ]
                      }
                    }
                  ]
                }
              }
            ]
          }
        }
      ]
    };

Woah! easy there, pilgrim! Let's break it down actually and take it from top to bottom...

1. It all must start with a fieldset (array). It's a wrapper property which contains further mapping declarations.

2. Then there's a fromEach (object) property following which represents a wrapper for arrays of objects.

3. The fromEach property must include at least a field (string) property, which declares the property in the source JSON object for get the value from. It always refers to an array type since it only may appear in the context of a fromEach block.

4. The to (string) propery may be found in a fromEach block, as well as in a fieldset. It defines the field in the target object where the value shall be writte to. It isn't mandatory if a from field is present, in case it's missing the field in the target object will have the same name as it's source counterpart.

5. The flatten(boolean) property is optional. It can be used in case the array the fromEach refers to shall be extracted from the array. That means that the properties of any objects are extracted and placed one level above. The flatten property is only valid within the scope of a single fromEach. That means other, nested fromEach blocks aren't affected. If such nested blocks must be "flattened" as well, the flatten property may be set to true again in their own context. The default for the "flattening" is false.

6. Then, there's fieldset again. Here the fieldset contains a precise mapping declaration for particular fields.

7. The from property denotes the key in the source object where the value shall be taken from. We can use chaining via dot (.) to cherry pick values out from nested object structures.

8. The to property is the key in the target object where the value shall be written to. It is not mandatory if a from property exists - in that case if it's not there, the default applies, which means that the 'write-to' property in the target will be the same as the 'from' from the source. Chaining can be applied here as well. It'll create a nested object structure with the last property to be the carrier of the value.

9. The toArray property defines whether the referenced value from the source shall be placed into an array in the target object. This might come handy when you want to perform further processing on the transformed json and need to have arrays for the particular properties.

10. The withTemplate property defines a template which may contain an arbitrary string with the possibility to embed references to props from the source. That comes handy if you want to construct a new property consisting of several fields from the source and maybe also some more text. In case the withTemplate prop is defined, the from property must not exist in the same scope, the two fields withTemplate and from are mutually exclusive. Another thing that changes if withTemplate is defined, is that the to property becomes mandatory and must be provided. This is because no implicit to field can be derived since the from property is not allowed in that case. Referenced properties may be nested, contain non-word characters, just like the props referenced by from are allowed to have.

11. It is possible to define a format for values in the target object. Currently this is only possible for dates. In general formatting can be declared by using the via property which is an object that holds the type of the value, the source format to parse from and the target one to re-format to. Formatting is also possible in combination with the withTemplate property, though it is only possible to define one formatting option for all referenced values in the template. If you perhaps have 2 date values referenced both will be re-formatted with the format defined in the via property.

That's a rather complex yet complete example since it makes use of the whole range of the currently implemented vocabulary of the DSL.