npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

safe-log-reader-enhanced

v1.0.8-1

Published

Safe Log Reader

Downloads

3

Readme

Build Status Code Coverage Code Climate NPM

Safe Log Reader

Read plain or compressed log files from disk, deliver as [batches of] lines to a log consumer. Wait for the log consumer to report success. Advance bookmark. Repeat ad infinitum.

Install

npm i safe-log-reader

Usage

const read = require('safe-line-reader');
read.createReader(filePath, {
    batchLimit: 1024,
    bookmark: {
        dir: path.resolve('someDir', '.bookmark'),
    }
})
.on('readable', function () { this.readLine(); })
.on('read', function (line, count) {
    // do something with this line of text
})
.on('end', function (done) {
    // close up shop and go home
});

Features

  • [x] Read plain text files
  • [x] Handles common log file events
    • [x] reads growing files (aka: tail -F)
    • [x] reads rotated logs
      • [x] reads the new file when it appears
        • [x] fs.watch tested on:
          • [x] Mac OS X
          • [x] Linux
          • [x] FreeBSD
      • [ ] continues reading old log file until quiet
    • [ ] file truncation (echo '' > foo.log)
    • [x] watches for non-existent log to appear
  • [x] Read compressed log files
    • [x] gzip (zlib)
    • [ ] bzip2
  • [x] Emits data as lines, upon request (paused mode streaming)
    • [x] Uses a Transform Stream to efficiently convert buffer streams to lines
    • [x] waits for confirmation, then advances bookmarks
  • [x] handles utf-8 multibyte characters properly
  • [x] Remembers previously read files (bookmarks)
    • [x] Perists across program restarts
      • [x] identifies files by inode
      • [x] saves file data: name, size, line count, inode
    • [x] When safe, uses byte position to efficiently resume reading
  • [ ] cronolog naming syntax (/var/log/http/YYYY/MM/DD/access.log)
    • [ ] watches existing directory ancestor
  • [ ] winston naming syntax (app.log1, app.log2, etc.)
  • [x] zero dependencies

Shippers

Similar Projects

  • tail-stream has good options for reading a file and handling rotation, truncation, and resuming. It had no tests so I wrote them and most have been merged. Bugs remain (demonstrated with Travis-CI test integration) unresolved. The author offered a license in exchange for the tests but the GPL is problematic.
  • tail-forever has character encoding detection and very basic file watching.
  • always-tail

The key "missing" feature of the node "tail" libraries is the ability to resume correctly after the app has stopped reading (think: kill -9) in the middle of a file.

Because files are read as [chunks of] bytes and log entries are lines, resuming at the files last byte position is likely to be in the middle of a line, or even splitting a multi-byte character. Extra buffered bytes not yet emitted as lines are lost, unless at restart, one rewinds and replays the last full $bufferSize. Then the consuming app needs to have duplicate line detection and suppression.

The key to resuming reading a log file safely is tracking line numbers and the byte steam offset the consuming app has committed. When saving bookmarks, the file position advances to the byte offset coinciding with the byte position of the last line your application has safely commited.

Safe-log-reader uses a Transform Stream to convert the byte stream into lines. This makes it dead simple to read compressed files by adding a .pipe(ungzip()) into the stream.

Copyright 2015 by eFolder, Inc.