Ship Postfix Logs to Elasticsearch

Parses postfix, rspamd, and rmilter log files and ships them to Elasticsearch as normalized documents.

Overview

Turns this:

Jul 26 04:18:34 mx12 postfix/pickup[20280]: 3mfHGL1r9gzyQP: uid=1208 from=<system>
Jul 26 04:18:34 mx12 postfix/cleanup[20659]: 3mfHGL1r9gzyQP: message-id=<[email protected]>
Jul 26 04:18:34 mx12 postfix/qmgr[28761]: 3mfHGL1r9gzyQP: from=<system>, size=813, nrcpt=1 (queue active)
Jul 26 04:18:34 mx12 postfix/smtp[20662]: 3mfHGL1r9gzyQP: to=<system>, relay=127.0.0.2[127.0.0.2]:25, delay=0.53, delays=0.13/0/0.23/0.16, dsn=2.0.0, status=sent (250 Queued! (#2.0.0))
Jul 26 04:18:34 mx12 postfix/qmgr[28761]: 3mfHGL1r9gzyQP: removed

Into this:

{
    "id": "3mfHGL1r9gzyQP",
    "host": "mx12",
    "events": [
      {
        "date": "2015-07-26T04:18:34-04:00",
        "action": "queued"
      },
      {
        "to": "system",
        "relay": "127.0.0.2[127.0.0.2]:25",
        "dsn": "2.0.0",
        "status": "sent (250 Queued! (#2.0.0))",
        "date": "2015-07-26T04:18:34-04:00"
      },
      {
        "date": "2015-07-26T04:18:34-04:00",
        "action": "removed"
      }
    ],
    "date": "2015-07-26T04:18:34-04:00",
    "isFinal": true,
    "uid": "1208",
    "message-id": "[email protected]",
    "from": "system",
    "size": "813",
    "nrcpt": "1",
    "delay": "0.53",
    "delays": "0.13/0/0.23/0.16"
  }

How it Works

• read a batch of log entries
• parse lines with postfix-parser
• fetch matching docs from Elasticsearch
• update/create normalized docs
• save new/updated docs to Elasticsearch

Install

npm i log-ship-elastic-postfix

Edit log-ship-elastic-postfix.ini, then launch with:

node server.js

With a custom config directory:

node server.js -config ~/etc/

Features

• [x] drop in modules for: reader, parser, and elasticsearch
• [x] official elasticsearch client load balances among ES hosts
• [x] config file is human friendly ini
• [x] can replay logs w/o duplicate ES documents
• [ ] streams multiple files simultaneously
• [ ] cronolog naming syntax (/var/log/http/YYYY/MM/DD/access.log)
  • [ ] watches existing directory ancestor
• [ ] winston naming syntax (app.log1, app.log2, etc.)
• [ ] email alerts for unrecoverable errors

Copyright 2015 by eFolder, Inc.