node-osm-stream

Overview

A fast and flexible NodeJS-based streaming parser for OpenStreetMap (.osm) files.

• Both incoming and outgoing streams available for piping from/to other streams
• Able to deal with large .osm files easily without loading the whole file onto memory
• Provided with the flexibility to work with and modify OSM objects (nodes, ways, relations) easily as it streams to you

Powered by node-expat for blazing fast parsing.

Table of Content

• Quick Start
  • Install
  • Basic example
  • More examples
• API
  • Class: OSMStream
  • Methods
  • Events
    • Events: 'node', 'way', 'relation'
    • Event: 'writeable'
    • Event: 'flush'
    • Events inherited from stream.Transform
• Test
• Known Issues
• Credits
• Links
• License

Quick Start

Install

npm install node-osm-stream

No fuss, no lints

Example: Reading from a local .osm file

var fs = require('fs');
var OSMStream = require('node-osm-stream');
var parser = OSMStream();

// open a local .osm filestream
fs.createReadStream('./path/to/file.osm')
	.pipe(parser);

parser.on('node', function(node, callback){
	// Modify current node object as you wish
	// and pass it back to the callback.
	// Or pass 'null' or 'false' to prevent the object being 
	// written to outgoing stream
	console.log(node);
	callback(node);
});

parser.on('way', function(way, callback){ callback(way); });

parser.on('relation', function(way, callback){ callback(relation); });

Easy-peasy, lemon-squeezy.

More examples

More advanced examples are available in the ./examples folder

Stream and format outgoing data to console (process.stdout)

Source: /examples/stream-to-stdout.js

To run example:

node ./examples/stream-to-stdout.js

Writing to a JSON file using Writeable steam (fs.createWriteStream)

Source: /examples/write-to-json.js

To run example:

node ./examples/write-to-json.js

API

Class: OSMStream

Class inherited from stream.Transform

Methods

All methods are inherited from stream.Transform

Events

Events: 'node', 'way', 'relation'

When an object (node/way/relation) from the .osm file has been parsed fully with its attributes and children (if any) ready, it will emit a 'node' or 'way' or 'relation' event, depending on the object type.

You can modify the outgoing data and passing it back to the callback. Or you can prevent the data from being passed downstream by passing back a null or false

It's important to note that since this is a streaming parser, any other objects (ways/relations) that may have referenced a skipped node may still hold its reference. It is up to the implementation to remove its references.

To see an example of a possible implementation, take a look at /examples/write-to-json.js

Note: If this event was registered, the callback must be passed back.

parser.on('node', function(node, callback) {
  // modify the node object as necessary and pass back to callback
  // or send a null or false to prevent it from going downstream
  callback(node);
});

parser.on('way', function(way, callback) {
  ...
  callback(way);
});
parser.on('relation', function(relation, callback) {
  ...
  callback(relation);
});

Event: 'writeable'

When a chunk of data is ready to be written to the outgoing stream, it will emit a 'writeable' event.

You can modify the outgoing data and passing it back to the callback. Or you can prevent the data from being passed downstream by passing back a null or false

Note: If this event was registered, the callback must be passed back.

parser.on('writeable', function(data, callback) {
  // there is some data to be passed to outgoing stream
  // modify 'data' as needed
  callback(data);
});

Event: 'flush'

After all the written data has been consumed through the outgoing stream, it will emit a 'flush' event. This will happened before the 'end' event is sent to signal the end of the readable side.

You can choose to pass in as much data as needed by passing it through the callback. Any data passed back will be written to the outgoing stream before the 'end' event is emitted.

Note: If this event was registered, the callback must be passed back.

parser.on('flush', function(callback) {
  var last_data = 'This is the last string sent to the outgoing stream';
  callback(last_data);
});

Events inherited from stream.Transform

In addition to the events above, the following are events inherited from stream.Transform class. Please refer to the offical documentation for more info: NodeJS API Documentation: stream.Transform

• Event: 'readable'
• Event: 'data'
• Event: 'end'
• Event: 'close'
• Event: 'error'
• Event: 'drain'
• Event: 'finish'
• Event: 'pipe'
• Event: 'unpipe'

Test

npm test

Known Issues

Credits

• Hafiz Ismail
• node-expat

Links

• twitter.com/sogko
• github.com/sogko
• medium.com/@sogko

License

Copyright (c) 2014 Hafiz Ismail. This software is licensed under the MIT License.