rxs-lib

All the stuff common to our Rethink projects

Google Authentication callback

const googleAuthCallback = require("rxs-lib/googleAuthCallback");
app.get("/auth/google/callback", passport.authenticate('google', { failureRedirect: '/login' }), googleAuthCallback(knex, authRouter.addToken));

JSON arrays to tabular data

This is used to turn JS object arrays to CSV ready arrays.

const tabulize = require("rxs-lib/tabulize");
const input = [
  { name: "John", age: 33 }, 
  { name: "Peter", age: 50 }
];
const output = tabulize(input);
//  [
//    ["name", "age"], 
//    ["John", 33], 
//    ["Peter", 50]
//  ]

Tabular data to CSV

The output of tabulize used here as input, to generate a CSV string

const csvify = require("rxs-lib/csvify");
const tabularData = [
  ["name", "age"], 
  ["John", 33], 
  ["Peter", 50]
];
const csv = csvify(tabularData);
// name,age
// "John",33
// "Peter",50

Math

Positive average

This function takes an array of numbers and calculates the average considering only the present (not null or undefined) positive (> 0) values

const { positiveAvg } = require("rxs-lib/math");

positiveAvg([2, 4, 6]); // 4
positiveAvg([0, 4, 6]); // 5
positiveAvg([0, 0, 8, 2]); // 5
positiveAvg([0, null, null, 8, 2]); // 5

JSX auth

Show / hide JSX components based on roles

import Auth from "rxs-lib/Auth";

<Auth authorizedRoles="role1,role2" userRoles={["role1"]}>
  <p>Private content</p>
</Auth>

** authorizedRoles can also be a string array *** if you don't provide authorizedRoles it will be always shown

Natural Language Processing (NLP)

Tokenization

const nlp = require("rxs-lib/nlp");

const tokens = nlp.tokenize("¡Y \ntambién     acentos, María!");

// tokens = ["y", "tambien", "acentos", "maria"]

N-gram matching

The emojis get sttriped away, so cannot be counted for the index location. Also might get consufed by very small words closely before the searched token.

const nlp = require("rxs-lib/nlp");

const indexes1 = nlp.approximateIndexesOf("Hard is hard, but you know that hard is hard, yes?", "hard is hard");
// indexes1 =  [0, 32]

const indexes2 = nlp.approximateIndexesOf("You cannot find only a piece of the sentence", "only the piece");
// indexes2 = []

const indexes3 = nlp.approximateIndexesOf("Hey, guys!!! Here we   are 😁, enjoying the first   words that came out 💤", "first words");
// indexes3 = [42]     Should be 43, but cannot count emojis

const indexes4 = nlp.approximateIndexesOf("A more simple example", "SimplE");
// indexes4 = [7]

const indexes5 = nlp.approximateIndexesOf("And, sometimes, an a-word is harder", "a-word");
// indexes5 = [16]    Should be 19, but the 'an' token gives a false positive

BigQuery data uploader

const buildBigQueryUploader = require("rxs-lib/bqUploader");
const bq = new bigquery.BigQuery({ projectId: "your-project-id" });

const bqUploader = await buildBigQueryUploader({ 
  bqClient: bq,
  chunkSize: 5000, // Inner use, you shouldn't need to use it
  datasetId: "your-dataset-id",
  id: "igv2-scraper", // Use any text that uniquely identifies this kind of task
  maxFileSizeInBytes: 1024 * 1024 * 500, // This would be 500 MB, by default it's 2 GB
  tableId: "your-table-id", 
  tempPath: "/home/user/temp", // Default .
  uniqueValidationFields: ["platform", "id"] // Duplicates validation **
});

await bqUploader.addItem({ name: "John", age: 33 });
await bqUploader.addItem({ name: "Susan", age: 32 });

// or...
await bqUploader.addItems([
  { name: "John", age: 33 },
  { name: "Susan", age: 32 }
]);

const totalItemsToUpload = await bqUploader.getTotalItemsToUpload();
// >> 2

const onTick = function onTick (totalItemsUploaded) {
  console.log(`Uploaded ${totalItemsUploaded} of ${totalItemsToUpload}`);
};

const uploadJobs = await bqUploader.upload(onTick);

Creates a bqUploader-files directory in the temporary path provided (or . by default), and saves the added items to NEWLINE_DELIMITED_JSON text files, with a maximum amount of chunkSize per file. When calling upload it first concatenates the written files into n files (limited in size by the maxFileSizeInBytes parameter); then it reads such files, one by one, and deletes all files after its successful upload.

IMPORTANT: Always prefer addItems over addItem, as each of these operations write to disk, which is a super costly operation.

Special case: single object immediate uploading

If you have only 1 item, and want to add and upload it immediatly, without building a batch, you can use this feature:

  // ...build uploader
  const uploadJobs = await bqUploader.upload({ name: "John", age: 33 });

Duplicates validation **

If you provide uniqueValidationFields parameter, when calling addItem it won't add any item that would be duplicated based on these fields. No error thrown, just doesn't add it.

Clearing items to upload

If you clear all items wrote to files for the given uploader will be physically deleted.

  // ...build uploader
  await bqUploader.clear();

IMPORTANT: Use only if you know what you're doing, and with extreme care (you might be deleting items queued to be uploaded later)