gumyen

Guess My Encoding. Small module that originated as a scratch to a personal itch around opening files which may be utf8/16, but which is now a more useful (though still small) utility for dealing with these encodings.

caveats

Currently this is optimised for the (rather specific) case of determining the encoding of XML and JSON files which may be any of UTF8/16-LE/16-BE. Gumyen works by reading the first 10 octets of a file and then (a) checking for a byte order mark, then if not found (b) checking for '{' encoded in utf16le or be as the first character of the file, and then (c) the string <?xml in each of the three encodings utf8, utf16le, utf16be. Additionally has utilities for stripping UTF BOM from buffers, strings and files, and writing strings and buffers to files with prepended BOMs.

usage

var gumyen = require('gumyen')

gumyen.encoding(filename, function(err, encoding) {
  // encoding is 'utf8', 'utf16le', 'utf16be'
});

var encoding = encodingSync(filename);
// sync version of above

gumyen.readFileWithDetectedEncoding(filename, function(err, data, encoding) {
  // encoding is 'utf8', 'utf16le', 'utf16be'
  // data is string contents of file
  // removes BOM
});

var data = gumyen.readFileWithDetectedEncodingSync(filename);
// sync version of above

var clean = gumyen.removeBOM(stringOrBuffer);

gumyen.writeFile(filename, data, options, function(err) {
  // wraps fs.writeFile, writes with BOM appropriate to encoding given in options
  // if options.writeBOM is truthy. data can be string or buffer
});

gumyen.writeFileSync(filename, data, options); {
// sync version of above

var bufferWithBOM = gumyen.addBOM(buffer, encoding);

dependencies

if you're wanting to work with utf16be, you'll need a module such as iconv-lite

install

npm install gumyen

test

npm test