osm-pbf-stream

OpenStreetMap Protocol Buffer Stream

Install via npm

$ npm install --save osm-pbf-stream

Index

• What
• Usage
• Speed

What

As OpenStreetMap's .osm.pbf files aren't pure Protocol Buffer messages, but are in a chunked, size delimited (and optionally, compressed) custom crafted format – this parses out the Protobuf message chunks.

So, osm.pbf goes in, pbf comes out.

Usage

var OsmPbf = require( 'osm-pbf-stream' )

var pbfStream = fs.createReadStream( 'berlin-latest.osm.pbf' )
  .pipe( new OsmPbf.BlobParser() )
  .on( 'header', ... )
  .on( 'blob', ... )

// Now you can process the Protocol Buffer stream
// with a streaming Protobuf parser, like `pbs`
var pbs = require( 'pbs' )
var osmSchema = fs.readFileSync( 'osmformat.proto', 'utf8' )
var messages = pbs( osmSchema )

var decoder = messages.PrimitiveBlock.decode()

decoder.primitivegroup( function( group, next ) {
  console.log( 'PrimitiveGroup.nodes', group.dense || group.nodes )
  next()
})

decoder.once( 'finish', function() {
  console.log( 'EOD' )
})

// Pipe the PBF stream to the decoder
pbfStream.pipe( decoder )

Speed

With the ~49 MB test file, it processes about 50 MB/s on a MacBook Pro.

BlobParser
  ✓ 1MB chunk size (986ms)
  ✓ Default (16KB) chunk size (1009ms)