json-dedupe

Installation

To use this module as a cli run:

npm i -g json-dedupe

For use as a npm module:

npm i json-dedupe

CLI Usage

Our CLI has a nice built in helper for the latest info and options, run:

$ json-dedupe -h
Usage: json-dedupe [options]

Options:
  -V, --version     output the version number
  -q --quiet        Toggle quiet mode
  -p --perf         Toggle performance output
  -d --data [data]  Point to the data to dedupe, either a json file or folder of json files
  -h, --help        output usage information

This list will always be up to date, and should be considered the source of truth for options.

output

If you point this tool to a folder or single file you will get an output of:

1. What was removed and the collision key
2. What remains

The output is formatted to match the initial input.

Module Usage

We've also included an export for node:

const {dedupe} = require('json-dedupe')
const data = require('./some-js-array-or-json')

dedupe(data, ['_id', 'email'], 'entryDate')

Documentation for Module

removeDuplicateByKey(valueMap, duplicateSet, uniqueKey, staleKey, value)

A helper to create a Map indexed on a supposed unique key, and add all objects that violate that unique key to a provided Set of values. This is an in-place transform.

Kind: global function

| Param | Type | Description | | --- | --- | --- | | valueMap | Map | the Map for all of your unique values | | duplicateSet | Set | the Set for any found duplicates | | uniqueKey | String | the key to find as a duplicate | | staleKey | String | the key to break a tie if a duplicate is found | | value | Object | the value to check |

dedupe(values, keys, staleKey) ⇒ Object

A helper to find all duplicates in an array based on _id and email. Information about end-state and what was removed is also logged out in the process.

expected format:

{
  _id: String,
  email: String,
  entryDate: String
}

Kind: global function Returns: Object - The deduped array and a map of the duplicates found by the key collision

| Param | Type | Description | | --- | --- | --- | | values | Array | An array of objects that match the expect object format | | keys | Array | An array of keys you want to dedupe on sorted by order of execution | | staleKey | String | The key for determining which entry is stale |