@liquicode/jsongin

Home: http://jsongin.liquicode.com

Version: 0.2.0

WARNING:

This version represents a significant divergence from previous versions of jsongin (v0.0.23 and older). Please review breaking changes in the Version History before replacing and upgrading.

A JSON Engine for MongoDB-Style Queries and Data Structure Manipulation

Quick Reference

• Library Guide
• Operator Reference
• Operator Authoring
• Document Manipulation
• Project History

Installation Guides

• NodeJS Usage
• Browser Usage

npm install --save @liquicode/jsongin

const jsongin = require( '@liquicode/jsongin' );

Overview

jsongin implements MongoDB's query, projection, update, and aggregation mechanics as ordinary Javascript functions over ordinary Javascript objects. There is no server, no connection, and no driver. You hand a function a document and a criteria, and it hands you back an answer.

Everything you pass to jsongin is itself a JSON document, and that is the idea the rest of the library is built on. A query is a document. So is a projection, an update, an aggregation pipeline, and an expression. Because your instructions are data rather than code, they can be stored in a database, sent across a wire, generated by another function, kept in a configuration file, and reviewed in a pull request like anything else you keep.

What jsongin Is Good For

• Working with data before it reaches a database. Filter, sort, project, and aggregate an in-memory array with the same criteria you would send to a MongoDB server - and send that same criteria to the server later, unchanged.

• Developing and testing without a database running. Work against an in-memory collection during development and switch to a real server for deployment without rewriting your queries. See @liquicode/jsonstor for storage adapters which carry one query interface across many platforms and mediums.

• Deciding things about a document. Query() answers true or false, which is what a permission check, a routing rule, a validation, or a feature flag needs. The rule is a document, so it can live next to the data it governs.

• Computing values out of documents. Evaluate() runs an aggregation expression against a document, so a derived field, a formatted label, or a computed total becomes a configuration value rather than a function somebody has to write and deploy.

• Recording and undoing change. Diff() describes what changed between two documents as an update document, and Invert() produces the update which undoes it - an audit trail and an undo stack out of the same pair of functions.

• Running work which has to survive being interrupted. See @liquicode/jsonproc, which builds a process runtime on this engine: a process is a JSON document and a run is a JSON value, so a half-finished job can be written to storage, moved to another machine, and picked up later.

• Shipping to the browser. One minified file, no dependencies, and the same behavior you get on the server.

How jsongin Is Organized

jsongin is a small set of functions, and a large set of operators which those functions read out of the documents you give them. Each kind of document you write is answered by its own family of operators:

| The document you write | The operators which read it | The functions which take it | |---|---|---| | A query - which documents match | Query Operators | Query(), Filter() | | A projection - which fields to keep | Projection Operators | Project() | | An update - how a document changes | Update Operators | Update(), Diff(), Invert() | | An expression - a value computed from a document | Expression Operators | Evaluate() | | A pipeline - stages a set of documents flows through | Stages and Accumulators | Aggregate() |

The families are not islands. The expression language is the same wherever it turns up: inside an $expr in a query, in a computed field of a projection, in an $addFields stage, and in a $do step of a jsonproc process, it is the same Evaluate(). Paths use MongoDB's dot notation everywhere, such as 'user.name', with no extensions of jsongin's own.

Every one of these families is MongoDB's, and jsongin implements 86.6% of them: 220 of the 254 operators MongoDB documents.

What is implemented is measured rather than asserted. Each implemented behavior is compared against a running MongoDB server, and the suite reports 100% agreement across 1054 compared behaviors. Run npm run parity-report for that number and npm run api-coverage for the surface numbers above.

The sections below introduce each of the main functions. See the Operator Reference for the full list of supported query, expression, update, stage, and accumulator operators.

The Main Functions

These two values are used by the examples throughout this section:

// A single document.
let document =
{
	id: 1001,
	user: { name: 'Alice', location: 'East' },
	profile: { login: 'alice', role: 'admin' },
	tags: [ 'Staff', 'Dept. A' ],
};

// A set of documents.
let players =
[
	{ team: 'red', name: 'Alice', points: 7, alive: true },
	{ team: 'red', name: 'Bob', points: 3, alive: true },
	{ team: 'blue', name: 'Carol', points: 9, alive: false },
];

Query( Document, QueryCriteria )

Tests a single document against a set of criteria and returns true or false. Field names use dot notation, and criteria may be combined with query operators.

jsongin.Query( document, { id: 1001 } ) === true
jsongin.Query( document, { 'user.name': 'Alice' } ) === true

// Several fields in one criteria must all match.
jsongin.Query( document, { tags: 'Staff', 'profile.role': 'admin' } ) === true

// Use operators for more than equality.
jsongin.Query( document, { 'profile.role': { $in: [ 'admin', 'super' ] } } ) === true
jsongin.Query( document, { $or: [ { 'user.location': 'East' }, { 'user.location': 'West' } ] } ) === true

See Query.

Filter( Documents, QueryCriteria )

Selects the documents in a set which match a query criteria. The criteria is exactly the one Query takes.

let alive = jsongin.Filter( players, { alive: true } );
// alive holds the Alice and Bob documents

let strong = jsongin.Filter( players, { points: { $gt: 5 } } );
// strong holds the Alice and Carol documents

See Filter.

Sort( Documents, SortCriteria )

Orders a set of documents. Use 1 to sort a field ascending and -1 to sort it descending. Sorting follows the MongoDB value ordering, so mixed types and missing fields have defined places.

jsongin.Sort( players, { points: 1 } );
// players is now ordered: Bob, Alice, Carol

jsongin.Sort( players, { team: 1, points: -1 } );
// players is now ordered: Carol, Alice, Bob

See Sort.

Distinct( Documents, DistinctCriteria )

Returns the distinct combinations of the named fields found in a set of documents.

let teams = jsongin.Distinct( players, { team: 1 } );
// teams is [ { team: 'red' }, { team: 'blue' } ]

let pairs = jsongin.Distinct( players, { team: 1, alive: 1 } );
// pairs is [ { team: 'red', alive: true }, { team: 'blue', alive: false } ]

See Distinct.

Project( Document, Projection )

Reshapes a document by including or excluding fields. A projection either names the fields to keep or the fields to remove, never both.

jsongin.Project( document, { id: 1, tags: 1 } );
// returns { id: 1001, tags: [ 'Staff', 'Dept. A' ] }

jsongin.Project( document, { profile: 0 } );
// returns the document without its profile field

// Nested fields can be named directly.
jsongin.Project( document, { id: 1, 'user.name': 1 } );
// returns { id: 1001, user: { name: 'Alice' } }

// A field whose value is an expression is computed.
jsongin.Project( { dmg: 12, armor: 5 }, { net: { $subtract: [ '$dmg', '$armor' ] } } );
// returns { net: 7 }

See Project.

Update( Document, Updates )

Applies MongoDB update operators to a document and returns the modified copy. The document you pass in is never changed.

jsongin.Update( document, { $set: { 'user.location': 'West' } } );
// the returned user is { name: 'Alice', location: 'West' }

jsongin.Update( document, { $set: { is_logged_in: true } } );  // adds a field
jsongin.Update( document, { $unset: { profile: '' } } );       // removes a field
jsongin.Update( document, { $push: { tags: 'New' } } );        // appends to an array

jsongin.Update( { n: 1 }, { $inc: { n: 5 } } );
// returns { n: 6 }

See Update.

Aggregate( Documents, Pipeline )

Runs a set of documents through an aggregation pipeline of stages.

jsongin.Aggregate( players,
	[
		{ $match: { alive: true } },
		{ $group: { _id: '$team', score: { $sum: '$points' } } },
	] );
// returns [ { _id: 'red', score: 10 } ]

jsongin.Aggregate( players,
	[
		{ $sort: { points: -1 } },
		{ $limit: 2 },
		{ $project: { _id: 0, name: 1 } },
	] );
// returns [ { name: 'Carol' }, { name: 'Alice' } ]

See Aggregate.

Evaluate( Document, Expression )

Computes a value from a document with an aggregation expression. A string beginning with $ is a reference to a field; anything else is a literal.

jsongin.Evaluate( { dmg: 12, armor: 5 }, { $subtract: [ '$dmg', '$armor' ] } ) === 7
jsongin.Evaluate( document, '$user.name' ) === 'Alice'

// The $expr query operator compares one field of a document to another.
jsongin.Query( { dmg: 12, armor: 5 }, { $expr: { $gt: [ '$dmg', '$armor' ] } } ) === true

See Evaluate.

Diff( Before, After )

Describes the change between two documents as an update document, in the same shape Update applies.

jsongin.Diff( { hp: 10, n: 1 }, { hp: 7 } );
// returns { $set: { hp: 7 }, $unset: { n: '' } }

See Diff.

Invert( Before, Patch )

Returns the update document which undoes a patch. It inverts any update document, not only the $set and $unset which Diff writes.

jsongin.Update( { hp: 10 }, { $inc: { hp: -3 } } );   // returns { hp: 7 }
jsongin.Invert( { hp: 10 }, { $inc: { hp: -3 } } );   // returns { $set: { hp: 10 } }

See Invert.

JSON Schema

Validate a document against a JSON Schema, in any draft from 4 to 2020-12, and learn why it failed. Write a schema from the documents you have, fill a document from a schema's defaults, or keep the fields a schema names. The query operator $jsonSchema reads a schema as MongoDB does.

let schema = { required: [ 'name' ], properties: { name: { type: 'string' }, age: { type: 'integer', minimum: 0 } } };

jsongin.ValidateDocument( { name: 'Alice', age: 30 }, schema );   // returns []
jsongin.ValidateDocument( { age: -1 }, schema ).length === 2

jsongin.InferSchema( [ { id: 1, tags: [ 'a' ] }, { id: 2, tags: [] } ] ).properties.tags;
// returns { type: 'array', items: { type: 'string' } }

jsongin.InitSchema( {}, { properties: { theme: { default: 'light' } } } );   // returns { theme: 'light' }
jsongin.Query( { name: 'Alice' }, { $jsonSchema: { required: [ 'name' ] } } ) === true

See the JSON Schema guide, ValidateDocument, InferSchema, InitSchema, and ProjectSchema.

Document Mechanics

Read, write, and reshape a document by path. A path is dot notation, the same notation the query and update functions use.

jsongin.GetValue( document, 'user.name' ) === 'Alice'
jsongin.GetValue( document, 'tags.0' ) === 'Staff'

jsongin.SetValue( document, 'user.tz', 'UTC' );      // creates the field
jsongin.DeleteValue( document, 'profile.login' );    // returns true when it removed something

// Flatten and Expand convert between a hierarchy and a set of paths.
jsongin.Flatten( { a: { b: 1 }, c: [ 1, 2 ] } );
// returns { 'a.b': 1, 'c.0': 1, 'c.1': 2 }

jsongin.Expand( { 'a.b': 1 } );
// returns { a: { b: 1 } }

// Merge combines two documents, descending only where both hold a sub-document.
jsongin.Merge( { a: 1, b: { x: 1 } }, { b: { y: 2 } } );
// returns { a: 1, b: { x: 1, y: 2 } }

See GetValue, SetValue, DeleteValue, Flatten, Expand, and Merge.

More Functions

Document Mechanics

• SplitPath( Path )
• JoinPaths( Path1, Path2, ... )
• GetValue( Document, Path )
• SetValue( Document, Path, Value )
• DeleteValue( Document, Path )
• Flatten( Document )
• Expand( Document )
• Hybridize( Document )
• Unhybridize( Document )
• Merge( DocumentA, DocumentB )
• Parse( JsonString, Options )
• Format( Value, Options )

JSON Schema

• ValidateDocument( Document, Schema, Options )
• InferSchema( Documents, Options )
• InitSchema( Document, Schema, Options )
• ProjectSchema( Document, Schema, Options )

Object Matching and Cloning

• LooseEquals( DocumentA, DocumentB )
• StrictEquals( DocumentA, DocumentB )
• CompareValues( ValueA, ValueB )
• Clone( Document )
• SafeClone( Document, Exceptions )

Data Types and Conversions

• ShortType( Value )
• BsonType( Value, ReturnAlias )
• AsNumber( Value )
• AsDate( Value )
• AsBoolean( Value )

Text Functions

A small set of string helpers is available at jsongin.Text: Compare, FindBetween, Matches, SearchReplace, and SearchReplacements.

See the Library Guide for more information.

Features

• MongoDB Compatibility:

  • 100% parity across 1054 compared behaviors, each one measured against a running MongoDB server.
  • 86.6% of the documented operator surface: 220 of 254 operators.
  • MongoDB's own path syntax, value ordering, and type rules, rather than an approximation of them.
  • Measure both numbers yourself with npm run parity-report and npm run api-coverage.

• Object Based Queries:

  • Compose queries in a structured and logical manner.
  • Easier to read, understand, and debug than a query string.
  • Maintain comments and documentation in your query source.
  • Programmatically create and structure data queries, store them, and send them across a wire.

• More Than Queries:

  • Describe a change between two documents with Diff(), and undo it with Invert().
  • Round-trip dates, regular expressions, and other typed values through storage with Format() and Parse().
  • Build a process runtime on top of it: see @liquicode/jsonproc.

• Developer Features:

  • No external dependencies. None. Zero.
  • 100% pure javascript, on the server and in the browser.
  • Single minified file (~267k, ~58k compressed) for web deployment.
  • Use the OpLog feature to help understand and debug queries.
  • Extend jsongin with operators of your own; no registry is private. See Operator Authoring.