Node-RED Contrib SimpleBU

SimpleBU is a custom Node-RED node that buffers, formats, and transmits measurement data to a server (e.g. deZem M2M endpoints).
It is designed for robustness in environments with unstable internet connections:

• If the server is reachable → data is sent immediately.
• If the server is not reachable → data is stored locally in a disk-based ring buffer.
• As soon as connectivity returns, the node will drain buffered data in chronological order.

This ensures no data loss, strict CSV format, and predictable uploads.

Quick Usage

Input Message Format

The node accepts Node-RED msg objects. The important fields are:

• msg.payload → numeric value(s)
• msg.topic → ID (required if payload is a single number)
• msg.timestamp (optional) → if not provided, the node auto-generates the current UNIX time.

Case 1: Single value

{
  "topic": "temp1",
  "payload": 22.5
}

Produces:

temp1,22.5,1757385119

Case 2: Multiple values (array)

{
  "payload": [
    ["temp1", 22.5],
    ["power", 120]
  ]
}

Produces:

temp1,22.5,1757385119
power,120,1757385119

Features

• CSV output format
  Each line is strictly id,value,timestamp.

• Timestamp handling

  • If msg.timestamp exists, it is used.
  • Supported formats:
    • UNIX seconds → 1757385119
    • UNIX milliseconds → 1757385119000
    • ISO8601 string → "2025-09-17T10:30:00Z"
    • JavaScript Date object
  • All are normalized to UNIX seconds.

• Single-file ring buffer

  • Configured by total size in MB (default 100 MB).
  • No splitting into multiple files.
  • When buffer is full → the oldest data is dropped (with a warning).
  • Prevents mixing data or losing order.

• HTTP POST uploads

  • Each request sends at most 100 KB.
  • Large files are split into 100 KB chunks.
  • Successfully sent chunks are removed from the file.

• Resilience

  • If server is down, data is kept locally until connectivity returns.
  • Buffered data is always sent oldest-first.

• Debug mode

  • If enabled, all errors and warnings are logged in Debug sidebar.
  • Examples:
    • “Missing ID (msg.topic)”
    • “Invalid value”
    • “Ring cap reached: dropped 512 KB”
    • “Cannot write to /data/buffer”

Configuration

| Field | Description | |-------------------------|-------------| | Endpoint-URL | Target URL where CSV data is POSTed. Required. Example: https://m2m.dezem.de/services/hpp/simple-bu.php. | | Simple SN | Serial number of the device/logger. Added to User-Agent header. | | Simple Type | Device type string. Added to User-Agent header. | | Ring Buffer Size (MB) | Maximum total size of the local buffer file (default: 100 MB). If exceeded, oldest data is dropped. | | Storage Path | Directory for storing the buffer file (e.g. /data/buffer). Must be writable. | | Flush Interval (s) | How often the node attempts to send data (in seconds). Default: 60s. | | HTTP Timeout (s) | Maximum time to wait for server response (in seconds). Default: 10s. | | Debug Enabled | If checked, detailed error/warning messages are shown in the Debug window. | | Name | Optional display name in the flow. |

Status Indicators

• 🔵 Buffered → Data accepted into RAM buffer.
• 🟢 Data sent → Data successfully uploaded.
• 🟢 Drained file → Buffered file chunk uploaded & removed.
• 🟡 Ring cap reached: dropped X KB → Oldest data was deleted because buffer exceeded configured size.
• 🔴 Send failed / Cannot write → Error writing to disk or sending to server.

Server Communication

• Data is sent via HTTP POST with headers:

  Content-Type: text/csv; charset=utf-8; header=absent
  User-Agent: IF:Simple TYPE:<SimpleType> SN:<SimpleSn>

• Server must respond with:
  • HTTP 2xx status
  • Response body: empty or OK

If not, the data remains in the buffer.

Error Handling

• Invalid ID or value → error status (red).
• Non-writable storage → “Cannot write to /path/to/storage”.
• Buffer overflow → “Ring cap reached: dropped N KB”.
• Send failed → data stays in buffer, retry later.

With Debug Enabled, full error details are logged.

License

MIT License
© 2025 deZem GmbH / Emir Dovletyarov