multipart-stream-js

Javascript library to parse an HTTP multipart stream.

What's a multipart stream for?

A multipart stream is a sequence of parts in one HTTP response, each part having its own headers and body. A stream might last forever, serving parts that didn't exist at the start of the request. This is a type of "hanging GET" or Comet request.

It's a simple HTTP/1.1 way of accomplishing what otherwise might require fancier server- and client-side technologies, such as:

• WebSockets
• HTTP/2 Server Push

Never-ending multipart streams seem popular in the IP camera space:

• Dahua IP cameras provide a multipart/x-mixed-replace stream of events such as motion detection changes. (spec)
• Hikvision IP cameras provide a multipart/mixed stream of events, as described here.
• wikipedia mentions that IP cameras use this format for MJPEG streams.

There's a big limitation, however, which is that browsers have fairly low limits on the number of concurrent connections. In Chrome's case, six per host.

I wrote this library as part of my own Moonfire NVR to implement live streams (a multipart stream of .mp4 media segments) and event streams. Due to the limitation above, I'm likely going to use WebSockets instead.

What is a multipart stream exactly?

A multipart response might look like this:

Content-Type: multipart/x-mixed-replace: boundary=B

--B
Content-Type: text/plain
Content-Length: 3

foo

--B
Content-Type: text/plain
Content-Length: 3

bar

and is typically paired with Transfer-Encoding: chunked or Connection: close to allow sending a response whose size is infinite or not known until the end.

I can't find a good specification. This WHATWG document describes multipart/x-mixed-replace loosely. It refers to RFC 2046 which defines multipart encodings originally used for rich emails. I don't think these HTTP multipart streams quite follow that RFC. My library currently requires:

• Content type multipart/...; boundary=.... In MIME media type terminology, the type is multipart; the subtype may be anything. There should be exactly one parameter, boundary.
• No preamble. That is, no arbitrary bytes to discard before the first part's boundary.
• Zero or more newlines (to be precise: \r\n sequences) between each part and the next part's boundary.
• A Content-Length line for each part. This is a much cleaner approach than producers attempting to choose a boundary that doesn't appear in any part and consumers having to search through the part body.
• No extra -- suffix on the final part's boundary. In practice, all the streams I've seen only end due to error, so this point has never come up.

Please open a github issue if you encounter a multipart stream which doesn't match these requirements.

What does this library do?

It reads from from a ReadableStream as defined in the WHATWG Streams API (spec, MDN) and turns it into another ReadableStream of parts. Each part has a header and body.

It works well with the WHATWG Fetch API (spec, MDN).

Example:

import multipartStream from `.../multipart-stream.js`;

async function processStream() {
  const response = await fetch(`http://example.com/stream`);
  const reader = multipartStream(
      response.headers.get('Content-Type'),
      response.body);
  while (true) {
    const {done, value} = reader.read();
    if (done) {
      break;
    }
    const {headers, body} = value;
    ...
  }
}

Where does it work?

Modern web browsers. It's tested on recent Chrome and Firefox via Karma and Jasmine.

It doesn't work on node.js, which lacks support for WHATWG Streams. I found a github project for support but it's just a skeleton.

It uses the npm ecosystem for package management.

Development notes

Contributions welcome. There's no CI setup (yet) but each commit should be tested via:

$ npm install
$ npm test
$ npm run lint

Please follow the Google Javascript style guide.

Author

Scott Lamb <[email protected]>

License

Your choice of MIT or Apache; see LICENSE-MIT.txt or LICENSE-APACHE, respectively.