@ndjson-utils/transform

A collection of functions to transform a NDJSON stream

Install

npm install @ndjson/transform

Usage

cat data.ndjson | <FUNCTION> <...ARGUMENTS>

count

Count the number of rows. Returns a single number.

filter

Takes one argument, the result of a (d, i) => function that returns a boolean.

If data.ndjson is:

{ "price": 14.55, "product": "A" }
{ "price": 6.95, "product": "B" }
{ "price": 18.15, "product": "C" }

cat data.ndjson | npx filter 'd.price > 10'

will return:

{ "price": 14.55, "product": "A" }
{ "price": 18.15, "product": "C" }

group

Takes one argument, the result of a (d, i) => function.

If data.ndjson is:

{ "price": 14.55, "product": "A" }
{ "price": 16.15, "product": "B" }
{ "price": 6.95, "product": "A" }
{ "price": 7.85, "product": "B" }

cat data.ndjson | npx group 'd.product'

will return:

[{"price":14.55,"product":"A"},{"price":6.95,"product":"A"}]
[{"price":16.15,"product":"B"},{"price":7.85,"product":"B"}]

and

cat data.ndjson | npx group 'i > 2'

will return:

[{"price":14.55,"product":"A"},{"price":16.15,"product":"B"},{"price":6.95,"product":"A"}]
[{"price":7.85,"product":"B"}]

map

Takes one argument, the result of a (d, i) => function.

If data.ndjson is:

{ "price": 14.55, "product": "A" }
{ "price": 6.95, "product": "B" }
{ "price": 18.15, "product": "C" }

cat data.ndjson | npx map '{...d, shop: "X", index: i }'

will return:

{"price":14.55,"product":"A","shop":"X","index":0}
{"price":6.95,"product":"B","shop":"X","index":1}
{"price":18.15,"product":"C","shop":"X","index":2}

num

Cast values to numbers. Useful after converting a csv file where all values are strings.

Takes one argument, a comma separated list of keys.

If data.ndjson is:

{ "amount": "3", "price": "3.45", "product": "A" }
{ "amount": "2", "price": "5.95", "product": "B" }

cat data.ndjson | npx num 'amount,price'

{"amount":3,"price":3.45,"product":"A"}
{"amount":2,"price":5.95,"product":"B"}

omit

To remove keys. Takes one argument, a comma separated list of keys.

If data.ndjson is:

{ "amount": "3", "price": "3.45", "product": "A" }
{ "amount": "2", "price": "5.95", "product": "B" }

cat data.ndjson | npx omit 'price,product'

returns

{"amount":"3"}
{"amount":"2"}

pairs

Returns an array of key value pairs.

If data.ndjson is:

{ "amount": "3", "price": "3.45", "product": "A" }
{ "amount": "2", "price": "5.95", "product": "B" }

cat data.ndjson | npx pairs

returns

[["amount","3"],["price","3.45"],["product","A"]]
[["amount","2"],["price","5.95"],["product","B"]]

pick

Pick keys. Takes one argument, a comma separated list of keys.

If data.ndjson is:

{ "amount": "3", "price": "3.45", "product": "A" }
{ "amount": "2", "price": "5.95", "product": "B" }

cat data.ndjson | npx pick 'price,product'

returns

{"price":"3.45","product":"A"}
{"price":"5.95","product":"B"}

reduce

If no arguments are passed, converts the NDJSON to a JSON array.

If data.ndjson is:

{ "amount": "3", "price": "3.45", "product": "A" }
{ "amount": "2", "price": "5.95", "product": "B" }

cat data.ndjson | npx reduce

returns

[{"amount":"3","price":"3.45","product":"A"},{"amount":"2","price":"5.95","product":"B"}]

First argument is the result of a (r, d, i) => function. Where r is the current value.

Second argument is the initial value, defaults to [].

If data.ndjson is:

{ "amount": 2, "price": 14, "product": "A" }
{ "amount": 1, "price": 16, "product": "B" }
{ "amount": 1, "price": 6, "product": "A" }
{ "amount": 3, "price": 7, "product": "B" }

cat data.ndjson \
  | node dist/reduce '{...r, [d.product]: (r[d.product] || 0) + d.amount * d.price}' '{}'

returns:

{"A":34,"B":37}

split

Turn a JSON file in an NDJSON stream.

Arguments:

• first argument, the JSON file
• second argument (optional) a dot separated path to the array to stream.

If data.json is:

[
  { "amount": 2, "price": 14, "product": "A" },
  { "amount": 1, "price": 16, "product": "B" },
  { "amount": 1, "price": 6, "product": "A" },
  { "amount": 3, "price": 7, "product": "B" },
]

npx split data.json

returns:

{ "amount": 2, "price": 14, "product": "A" }
{ "amount": 1, "price": 16, "product": "B" }
{ "amount": 1, "price": 6, "product": "A" }
{ "amount": 3, "price": 7, "product": "B" }

If data.json is:

{
  "results": [
    {
      "sales": [
        { "amount": 2, "price": 14, "product": "A" },
        { "amount": 1, "price": 16, "product": "B" },
        { "amount": 1, "price": 6, "product": "A" },
        { "amount": 3, "price": 7, "product": "B" },
      ]
    }
  ]
}

npx split data.json results.0.sales

returns:

{ "amount": 2, "price": 14, "product": "A" }
{ "amount": 1, "price": 16, "product": "B" }
{ "amount": 1, "price": 6, "product": "A" }
{ "amount": 3, "price": 7, "product": "B" }

sum

Takes one argument, the key to sum.

If data.ndjson is:

{ "amount": 2, "price": 14, "product": "A" }
{ "amount": 1, "price": 16, "product": "B" }
{ "amount": 1, "price": 6, "product": "A" }
{ "amount": 3, "price": 7, "product": "B" }

cat data.ndjson | npx sum amount

returns 7

uniq

If no arguments are passed, returns unique rows.

If data.ndjson is:

{ "amount": 2, "product": "A" }
{ "amount": 1, "product": "B" }
{ "amount": 2, "product": "A" }
{ "amount": 3, "product": "B" }

cat data.ndjson | npx uniq

returns:

{"amount":2,"product":"A"}
{"amount":1,"product":"B"}
{"amount":3,"product":"B"}

If an argument is provided as the result of a function (d, i) =>, rows on which the value has already been returned are ignored.

On the same data.ndjson,

cat data.ndjson | npx uniq 'd.product'

will return:

{"amount":2,"product":"A"}
{"amount":1,"product":"B"}

values

Returns an array with the values of the row.

If data.ndjson is:

{ "amount": 2, "product": "A" }
{ "amount": 1, "product": "B" }
{ "amount": 2, "product": "A" }
{ "amount": 3, "product": "B" }

cat data.ndjson | npx values

returns:

[2,"A"]
[1,"B"]
[2,"A"]
[3,"B"]