@event-horizon-data/agent

The Event Horizon Collection Agent — a small Node daemon that tails log files and ships each line, typed, to a ProbeNode. No YAML pipelines: two flags and a path.

Quickstart

npx @event-horizon-data/agent \
  --endpoint https://ingest.example.com/ingest/probe/<clientId>/<nodeId> \
  --key ehp_xxxxxxxx \
  --tail /var/log/app.log

Add more --tail <path> flags for more files.

Single-file binary (no Node required)

Download one file, make it runnable, run it — nothing to install:

./eh-agent \
  --endpoint https://ingest.example.com/ingest/probe/<clientId>/<nodeId> \
  --key ehp_xxxxxxxx \
  --tail /var/log/app.log

./eh-agent --version and ./eh-agent --help work standalone.

Building the binary

npm run build:sea    # produces ./eh-agent for the current OS/arch

This uses Node's official Single Executable Applications (SEA): the agent — including its own copy of Node — is packed into one file. The only build-time tool is postject (a devDependency); there is no new runtime dependency.

Other operating systems

A SEA binary is built for the OS/arch of the machine that builds it, and SEA cannot cross-compile. To ship macOS/Windows/arm binaries, run npm run build:sea on each target platform (typically a CI build matrix).

Code signing

macOS and Windows require the executable to be re-signed after injection or the OS may refuse to run it (codesign on macOS, signtool on Windows). This is a manual/CI step per platform and is not performed by build:sea.

Config file (optional, for the service case)

endpoint = "https://ingest.example.com/ingest/probe/<clientId>/<nodeId>"
key = "ehp_xxxxxxxx"
state_dir = "/var/lib/eh-agent"
batch_size = 50

[[tail]]
path = "/var/log/app.log"
[[tail]]
path = "/var/log/nginx/access.log"

eh-agent --config /etc/eh-agent.toml

Flags override file values (CLI flag > TOML > default).

Running as a systemd service

Generate a ready-made, hardened unit and install it:

eh-agent --systemd-unit | sudo tee /etc/systemd/system/eh-agent.service
sudo useradd --system --no-create-home --shell /usr/sbin/nologin eh-agent
sudo systemctl enable --now eh-agent

The agent then starts on boot and restarts on failure. Checkpoints persist in the systemd-managed /var/lib/eh-agent. To read root-owned logs under /var/log, add your distro's log group to the unit (SupplementaryGroups=adm on Debian/Ubuntu, SupplementaryGroups=systemd-journal on RHEL/Fedora) and run systemctl daemon-reload.

If your config lives elsewhere, pass it through — eh-agent --config /path/to.toml --systemd-unit puts that path in the generated ExecStart.

Database changes (CDC)

The agent can read the change log of a database inside your network and send the changes to Event Horizon, so the database never has to be opened to the internet. Only outbound HTTPS leaves your network, and the database password never leaves your machine.

In the app, pick an engine (MySQL, PostgreSQL, SQL Server or MongoDB) and the table on the probe node; generate the probe's key and copy its ingest URL. Then, on a machine that can reach the database, one line:

EH_CDC_PASSWORD=... eh-agent \
  --endpoint https://ingest.example.com/ingest/probe/<clientId>/<nodeId> \
  --key ehp_xxxxxxxx \
  --cdc mysql://[email protected]/shop

--cdc takes mysql://, postgres://, mongodb:// or sqlserver://. The password is read from EH_CDC_PASSWORD, never from the command line (it would show in the process list). The usual query-string options work (?tls=true, MongoDB's ?replicaSet=rs0&authSource=admin, SQL Server's ?trustServerCertificate=true). Add --cdc-tls for TLS, --cdc-never "customer_email,tax_id" to keep columns from ever leaving your network, and --cdc-allow "orders,payments" if you want the agent to refuse any table that is not on your list, whatever the app asks for (without it the agent reads the one table the app picked, which the database user's own grants already bound).

For several tables (one probe per table), or other options, use [[cdc]] blocks in the config file instead:

[[cdc]]
endpoint = "https://ingest.example.com/ingest/probe/<clientId>/<nodeId>"
key_env  = "EH_PROBE_KEY"          # the probe key, from the environment (or key_file)
engine   = "mysql"                 # mysql | postgres | mongodb | sqlserver
host     = "10.0.3.12"             # a private address is fine here
database = "shop"
user     = "eh_cdc"
password_env = "EH_CDC_PASSWORD"   # the default; or password_file. Never written in this file
tls      = true
allow_tables  = "orders, payments" # optional: refuse any other table
never_columns = "customer_email, tax_id"

Other options: port, tls_ca (PEM file), trust_server_certificate (SQL Server), auth_source and replica_set (MongoDB; a MongoDB without authentication needs no user).

• You keep the last word. The app chooses the table and the columns to leave out; --cdc-never / never_columns are removed on top of that, here, before anything leaves your network, and --cdc-allow / allow_tables make the agent read nothing that is not listed.
• Nothing is lost. The same delivery contract as the cloud change feed: at least once, with deterministic ids, and the position only advances after Event Horizon confirmed it stored the changes. The position is kept on the Event Horizon side, so reinstalling the agent or moving it to another machine does not copy the table again.
• One agent per probe. Starting a second agent with the same probe takes over; the first one stands down and says so.
• The database needs its change log on. The agent tells you in words what is missing (MySQL: binlog_format=ROW; PostgreSQL: wal_level=logical; SQL Server: CDC enabled on the table; MongoDB: a replica set). Use a read replica where you have one.
• PostgreSQL keeps a replication slot while the agent is away. If you decommission the agent for good, drop it (SELECT pg_drop_replication_slot('eh_cdc_<nodeid>')), or the database keeps its log.

What lands in your Table

Each line becomes a record. If the line is JSON, its keys become columns; otherwise the line lands as message. The agent always attaches host, path, and offset (these win over any same-named field in the log), and an idempotent event_id. timestamp is stamped by the server on receipt.

Event time (date normalization)

Point --timestamp-field at the field that carries the log's own time and the agent adds a canonical event_time (ISO 8601, UTC) so you can chart by when the event happened, not just when it arrived:

eh-agent --endpoint <url> --key <key> --tail /var/log/app.log \
  --parse '(?<ts>\S+) (?<message>.*)' \
  --timestamp-field ts

Formats are auto-detected (ISO 8601, epoch seconds/ms, syslog Mmm dd HH:MM:SS, common-log dd/Mmm/yyyy:HH:MM:SS ±zzzz, and yyyy-mm-dd HH:MM:SS), or pin one with --timestamp-format. For formats without a timezone, --timestamp-tz (default UTC) says how to interpret them. This is non-destructive: the original field and the server's arrival timestamp are left untouched, and an unrecognised value simply leaves event_time unset — the line is never dropped.

Delivery guarantees

• File sources (--tail, --watch): at-least-once always (never loses after a crash), and exactly-once across crashes — each line's event_id is derived from its file identity and byte offset, so a re-read after a crash carries the same id and the server deduplicates it.
• Syslog (--syslog-udp/--syslog-tcp): every received message is written to an on-disk spool before being shipped, so a crash does not lose it, and each spooled record ships with a position-derived event_id — giving syslog the same exactly-once across crashes guarantee for everything that reached the spool. By default the spool is flushed to disk in small groups (--spool-fsync batch), so the only thing a crash can lose is the last split-second of just-received messages not yet flushed; pass --spool-fsync always to flush every message immediately (zero loss, lower throughput). The spool lives under the state directory, is split into segments, and each segment is deleted once fully shipped; if the destination is unreachable long enough for the spool to reach its size limit, the agent pauses intake rather than filling the disk.

Rotation

Handles log rotation by dev+inode: rename-then-create (drains the old file, then follows the new one from the start) and truncate-in-place.

Running as a service

Point systemd at eh-agent --config /etc/eh-agent.toml — or generate a ready-made hardened unit with eh-agent --systemd-unit (see Running as a systemd service).

Watching a directory (rsync / spool dirs)

Directory watching detects new and changed files with low latency: the agent uses fs.watch/inotify to react the instant a file lands, backed by a periodic scan so nothing is ever missed (if the kernel notifier is unavailable it transparently falls back to plain polling). Add --watch-recursive to watch subdirectories too:

eh-agent --endpoint <url> --key <key> --watch /var/spool/incoming --glob '*.log' --watch-recursive

Files are still only ingested once their size is stable, so a file being written is never read half-complete.

Instead of tailing a fixed file, watch a directory for files that land:

eh-agent --endpoint <url> --key ehp_xxx \
  --watch /var/spool/in --glob '*.log' --mode content --on-complete move --archive-dir /var/spool/done

• --mode content (default): each newly-landed, stable file is read to EOF and its lines shipped as records — same typed-record model as --tail.
• --mode metadata: emit one file-integrity event per created/modified file ({ path, size, mtime, event, host, event_id }, plus sha256 if --hash). No content is read (unless --hash).
• --on-complete (content only): leave (default, never deletes), delete (after every line is acked), or move (into --archive-dir).
• Discovery is by polling; in-progress rsync temp files (.name.XXXXXX, *.tmp, *.partial) are ignored, and a file is only processed once its size is stable across a poll — so partial transfers are never read.
• A single line whose serialised record exceeds the ~1 MB wire body cap is silently skipped with an onError (never a 413 or permanent stop); with on_complete: delete/move the file is still disposed after the remaining lines are acked — an oversize line is dropped, not retried.

TOML form (tail and watch can coexist):

[[tail]]
path = "/var/log/app.log"

[[watch]]
dir = "/var/spool/in"
glob = "*.log"
mode = "content"
on_complete = "move"
archive_dir = "/var/spool/done"

Receiving syslog (UDP / TCP)

Run the agent as a syslog receiver — other hosts/devices send to it, and each message lands typed:

eh-agent --endpoint <url> --key ehp_xxx --syslog-udp 5514 --syslog-tcp 5514 --syslog-bind 0.0.0.0

• Parses RFC 5424 (modern, with structured-data exploded into sd.<id>.<param> columns) and RFC 3164 (BSD) into typed fields — facility, severity, timestamp, host, app, procid, message — falling back to { message } for anything else.
• The message's own host (the sending device) is preserved; the agent adds received_by, peer (sender IP), transport, and received_at.
• Reliability by transport: TCP applies backpressure (it stops reading when delivery is backed up, so the sender blocks — no loss); UDP uses a bounded buffer and drops the oldest under sustained overload (inherent to UDP syslog).
• Use a port >1024 (e.g. 5514) unless running as root; :514 is privileged.

TOML form (coexists with tail/watch):

[[syslog]]
udp_port = 5514
tcp_port = 5514
bind = "0.0.0.0"

Encrypted syslog (TLS)

Receive syslog over TLS (RFC 5425) instead of, or alongside, plaintext UDP/TCP:

eh-agent \
  --endpoint https://ingest.example.com/ingest/probe/<clientId>/<nodeId> \
  --key ehp_xxxxxxxx \
  --syslog-tls 6514 \
  --syslog-tls-cert /etc/eh-agent/server.pem \
  --syslog-tls-key  /etc/eh-agent/server.key

The agent presents server.pem to connecting clients (TLS ≥ 1.2). Records that arrive encrypted are tagged transport: "tls".

Mutual TLS (verify the sender): add a CA and require client certificates — the agent then only accepts clients presenting a certificate signed by that CA:

  --syslog-tls-ca /etc/eh-agent/ca.pem \
  --syslog-tls-client-auth

TLS syslog records go through the same durable spool and exactly-once delivery as every other source.

Parsing lines into typed columns (--parse)

Turn an unstructured line into columns with one pattern — a regex with named groups, sugared with a few grok aliases. Works on a tailed/watched line and on a syslog message body:

eh-agent --endpoint <url> --key ehp_xxx --tail /var/log/nginx/access.log \
  --parse '%{IP:client} \S+ \S+ "%{WORD:method} %{PATH:route}[^"]*" %{INT:status} %{INT:bytes}'

• Aliases: %{INT} %{NUMBER} (numeric — coerced to a number) %{WORD} %{NOTSPACE} %{DATA} %{GREEDYDATA} %{IP} %{HOSTNAME} %{PATH} %{QS} %{TIMESTAMP_ISO8601} %{LOGLEVEL}. Name a capture as %{ALIAS:name}; raw (?<name>…) works too.
• Each named group becomes a column; %{INT}/%{NUMBER} groups land as numbers. A group named timestamp becomes the event timestamp (passed through as-is — the server parses it).
• The original message is kept — unless your pattern names a group message, which replaces it. Agent provenance (host/path/…) always wins over a parsed group of the same name.
• %{INT} and %{NUMBER} values are coerced with JavaScript's Number, so integers beyond ~2^53 may lose precision; keep them as strings (omit the :name numeric alias) if exact large integers matter.
• A line that doesn't match is shipped unchanged (parse is best-effort).
• Per-source in TOML:

[[tail]]
path = "/var/log/nginx/access.log"
parse = '%{IP:client} \S+ \S+ "%{WORD:method} %{PATH:route}[^"]*" %{INT:status}'

[[syslog]]
udp_port = 5514
parse = 'pool=%{INT:pool} %{GREEDYDATA:detail}'

Your regex is your responsibility (as in Logstash/Vector) — the agent applies it per line (already capped at the max line size), with no engine timeout.

Keeping multi-line events together (--multiline)

A stack trace spans many lines but is ONE event. Glue them into a single record with a boundary rule:

eh-agent --endpoint <url> --key ehp_xxx --tail /var/log/app.log \
  --multiline '^\d{4}-\d\d-\d\d' --multiline-mode start

• --multiline-mode start (default): a line MATCHING the regex starts a new event; every other line appends to it (a timestamped line starts, the indented stack-trace lines continue).
• --multiline-mode continue: a line MATCHING the regex is a continuation (appends to the previous); a non-matching line starts a new event (e.g. --multiline '^\s' — continuation lines begin with whitespace).
• The joined text is the physical lines separated by \n, shipped as one record's message (then a --parse pattern, if set, runs on the whole event).
• A live log's last event is flushed after a few seconds of quiet (multiline_timeout_ms, default 5000); a completed file flushes its last event immediately; a run-away pattern is capped at multiline_max_lines (default 500).
• Additional CLI flags: --multiline-timeout <ms> (default 5000) and --multiline-max-lines <n> (default 500) are also available to tune timeouts and buffer caps.

TOML (tail/watch only; syslog is already one message per record):

[[tail]]
path = "/var/log/app.log"
multiline = '^\d{4}-\d\d-\d\d'
multiline_mode = "start"
multiline_timeout_ms = 5000
multiline_max_lines = 500

Delivery guarantee — never loses, and (for files) never duplicates

The agent is at-least-once — it never loses data. For file sources (--tail, --watch) it is additionally exactly-once across crashes: each record's event_id is a deterministic fingerprint of the file plus the line's byte offset, so if the agent crashes in the window between a batch being accepted and its progress being saved, the re-read produces the same event_ids and the server deduplicates them — no duplicate slips through.

Syslog (--syslog-udp/--syslog-tcp) now achieves exactly-once across crashes via an on-disk spool. Every received message is written to the spool before shipment; a crash does not lose it. By default the spool is flushed in batches (--spool-fsync batch), so the only loss window is the last split-second of just-received messages not yet flushed to disk; pass --spool-fsync always to flush every message immediately (zero loss in this window, but lower throughput). The spool is split into segments that are deleted once fully shipped; if the destination backs up long enough for the spool to fill, the agent pauses intake rather than dropping data.