MindMeld Poster

This posts documents to the MindMeld API. The documents should be in files or a stream, one JSON object per line.

To set up, make sure you have node installed, and run npm install in this directory.

Command Line Usage

The poster executable lives at bin/mindmeld-poster. To install the module globally and add this executable to your path, run:

npm install -g mindmeld-poster

If you are using this module as a dependency for another project, you can use {PROJECT_ROOT}/node_modules/.bin/mindmeld-poster.

To view the help:

mindmeld-poster --help

To read a file and post its contents, you can specify the path to the file an admin token to use to authorize your request to the API and domainid of the domain to which you will post:

mindmeld-poster -t <admin token> -d <domainid> <file>

To crawl a directory, posting the contents of all contained files:

mindmeld-poster -t <admin token> -d <domainid> <dir>

To read from stdin:

cat myfile | mindmeld-poster -t <admin token> -d <domainid>

If you haven't installed this package globally, you can still run commands from the root of this repository by giving the path to the command-line script:

./bin/mindmeld-poster -t <admin token> -d <domainid> <file>

If you just want to see the documents found in your file/directory without posting them to the command line, you can specify -O (note that you don't even need an admin token or domainid to do this):

mindmeld-poster -O <file>

By default, mindmeld-poster will try to concurrently post 5 documents to the MindMeld API concurrently. To change this number, you can add a -c option:

mindmeld-poster -t <admin token> -d <domainid> -c 10 <file>

Programmatic Usage

You can use this as a node module. To include it, just require it in your node.js javascript source as follows:

var poster = require('mindmeld-poster');

poster has several useful functions; the functions take an optional options object and a callback:

// Function called when errors are encountered, when each document is posted,
// and when the poster has completed.
var callback = function(err, response, finished) {
  // err: any error that was encountered, usually null
  // response: the response from the MindMeld API
  // finished: true when all documents have been processed, false otherwise
};

var options = {
  suppressLogs: false, // If true, suppress all output to stdout and stderr
  logger: null // A custom logger. The logger must implement `debug()`, `log()`, `info()`, `warn()`, `error()`, and `fatal()`.
  toStdout: false, // If true, print documents to stdout instead of posting.
  server: 'https://mindmeldv2.expectlabs.com', // URL of server to post to.
  token: MM_API_TOKEN, // Admin MM Token
  domainid: DOMAIN_ID, // domainid of domain where documents will be posted
  concurrent: 5, // Max number simultaneous API connections
  transform: function (doc) { return newDoc; }, // Function to map objects through
  filter: function (doc) { return shouldKeepDoc; } // Function to filter objects
};

To post a file:

poster.postFiles([pathToFile1, pathToFile2, ...], options, callback);
poster.postFiles(singleFilePath, options, callback);

To post a dir:

poster.postDir(pathToDir, options, callback);

To post a Readable stream:

poster.postStream(readableStream, options, callback);