node-red-contrib-compression

A collection of Node-RED nodes for time-series data compression, caching, visualization, and data processing utilities.

Features

• Time-series compression with multiple algorithms (Swinging Door, Exception-based, Deduplication)
• Message caching with TTL and size limits
• State sampling for periodic data snapshots
• Data visualization with real-time charts
• Type routing for automatic data type detection and routing
• ID-to-name mapping with web-based table editor
• Signal simulation for testing compression algorithms

Installation

npm install node-red-contrib-compression

Or install directly from the Node-RED palette manager.

Nodes

Compression

Time-series data compression node that reduces storage and transmission overhead while preserving data fidelity.

Algorithms:

• Deduplicate - Removes consecutive duplicate values
• Deduplicate+prev - Deduplication with previous point insertion on value change
• Exception - Outputs when value changes by more than a deviation threshold
• Exception+prev - Exception-based with previous point insertion
• Swinging Door - Swinging door trending algorithm for optimal linear compression
• Timedelta - Simple time-based sampling at regular intervals

Configuration:

• algorithm - Compression algorithm to use
• deviation - Deviation threshold (for exception and swinging-door algorithms)
• minDuration - Minimum time between outputs (seconds)
• maxDuration - Maximum time between outputs (seconds)
• timestampField - Message property containing timestamp (default: timestamp)
• valueField - Message property containing value (default: payload)
• snapshotKey - Global context key for state storage (default: snapshot)

Outputs:

1. Original message (always passed through)
2. Compressed message (only when point should be stored) or null

Example:

// Input message
{
  topic: "temperature",
  timestamp: 1609459200000,
  value: 23.5
}

// Compressed output includes
{
  ...originalMessage,
  compressed: true,
  skippedCount: 5  // Number of points skipped since last output
}

Cache

Stores messages in flow or global context with optional TTL and size limits. Useful for maintaining current state of multiple data points.

Configuration:

• contextName - Context variable name (default: cache)
• contextType - global or flow
• keyProperty - Message property to use as cache key (default: topic)
• ttl - Time-to-live in seconds (0 = no expiration)
• maxSize - Maximum number of cached entries (0 = unlimited, uses LRU eviction)
• outputOnUpdate - Whether to pass messages through on first output

Special Commands:

• msg.topic = "_cache" - Retrieve entire cache
• msg.topic = "_get" with msg.key - Get specific entry
• msg.topic = "_clear" - Clear all cached data

Example:

// Store message
{ topic: "sensor1", payload: 42.5 }

// Retrieve all cached values
{ topic: "_cache" }
// Returns: { payload: { sensor1: {...}, sensor2: {...} } }

Sample

Reads all entries from a global context dictionary and outputs them as individual messages or bulk array. Works perfectly with the Cache node to periodically sample current state.

Configuration:

• stateKey - Global context key to read from (default: cache)
• outputMode - individual (one message per entry) or bulk (single array message)

Example:

// Trigger sampling
{ payload: "trigger" }

// Individual output mode sends multiple messages:
{ topic: "sensor1", payload: {...}, _sampledAt: 1609459200000 }
{ topic: "sensor2", payload: {...}, _sampledAt: 1609459200000 }

// Bulk output mode sends one message:
{ payload: [{...}, {...}], topic: "bulk", count: 2 }

Chart

Real-time visualization node that displays time-series data in the Node-RED editor. Automatically separates raw and compressed data into different series.

Configuration:

• title - Chart title
• maxPoints - Maximum points per series (default: 100)
• timestampField - Message property containing timestamp (default: timestamp)
• valueField - Message property containing value (default: payload)
• seriesField - Message property for series name (default: topic)

Features:

• Automatically creates separate series for raw vs compressed data
• Real-time updates via WebSocket
• Clear chart via msg.topic = "_clear"
• HTTP endpoints for data access and clearing

Type Router

Routes messages based on payload data type with automatic type coercion from strings.

Outputs:

1. Integer values
2. Float values
3. Boolean values
4. String values

Features:

• Converts string representations ("true", "123", "45.67") to appropriate types
• Adds msg.dataType property indicating detected type
• Useful for processing data from text-based protocols (MQTT, HTTP, etc.)

Example:

// Input: { payload: "123.45" }
// Output on port 2 (float): { payload: 123.45, dataType: "float" }

// Input: { payload: "true" }
// Output on port 3 (bool): { payload: true, dataType: "bool" }

Mapping

ID-to-name mapping table with a web-based editor. Creates bidirectional lookup dictionaries in flow or global context. Useful for tag databases, sensor mappings, and configuration management.

Configuration:

• tableName - Display name for the table
• contextName - Context variable name for dictionaries
• contextType - global or flow

Features:

• Persists mappings to JSON files in {userDir}/mapping-data/
• Web-based table editor (click button in node editor)
• Auto-save on edit
• Copy/paste TSV format (Excel compatible)
• Creates forward lookup: { id → {name, type, deviation, interval} }
• Creates reverse lookup: { name → {id, type, deviation, interval} }

Outputs:

1. ID field
2. Name field
3. Full object with metadata

Commands:

• msg.payload = "get" - Output all mappings
• msg.payload = "clear" - Clear all mappings
• { id: "...", name: "...", type: "...", deviation: 0, interval: 0 } - Add single mapping
• msg.payload = [...] - Replace all mappings with array

Example Usage:

// Access mapping in function node
const mapping = global.get("mapping"); // or flow.get()
const metadata = mapping["sensor123"];
// Returns: { name: "Temperature Sensor", type: "ExcDev", deviation: 0.5, interval: 60 }

// Reverse lookup
const reverseMapping = global.get("mapping_reverse");
const metadata = reverseMapping["Temperature Sensor"];
// Returns: { id: "sensor123", type: "ExcDev", deviation: 0.5, interval: 60 }

Simulator

Generates simulated signal data for testing compression algorithms.

Waveforms:

• Sinusoid - Smooth sine wave oscillating between -amplitude and +amplitude
• Seesaw - Sawtooth wave with linear rise from -amplitude to +amplitude
• Rectangle - Square wave alternating between -amplitude and +amplitude
• Random - Random values uniformly distributed between -amplitude and +amplitude

Configuration:

• waveform - Signal type to generate
• amplitude - Peak deviation from offset value
• frequency - Oscillation frequency in Hz (cycles per second)
• offset - Center value around which signal oscillates
• interval - Time between samples in milliseconds
• topic - Value for msg.topic in output messages
• autoStart - Start generating signals when flow deploys

Control:

• msg.payload = "start" or true - Start continuous generation
• msg.payload = "stop" or false - Stop generation
• msg.payload = "reset" - Reset time counter to zero
• Any other value - Generate single sample

Output:

{
  topic: "simulation",
  payload: 1.234,
  timestamp: 1609459200000
}

Example Flows and Test Cases

Comprehensive test cases and example flows are available in the examples folder:

• examples.json - Original example flows for all nodes
• all-nodes-test.json - Complete test suite with 8 test scenarios
• test-cache.json - Focused cache node tests (4 scenarios)
• test-simulator.json - Focused simulator tests (6 scenarios)
• README.md - Detailed test documentation and usage instructions

Import any of these files in Node-RED to see working examples and test all functionality.

Typical Use Case

A common pattern for industrial IoT or sensor data processing:

[Simulator] → [Compression] → [Cache] → [Chart]
                   ↓              ↓
              [Database]     [Sample] → [Dashboard]

1. Simulator - Generate test signals (sinusoid, sawtooth, square, random)
2. Compression - Reduce data volume while preserving trends
3. Cache - Maintain current state of all sensors
4. Chart - Visualize raw vs compressed data
5. Sample - Periodically snapshot current state for dashboards
6. Database - Store only compressed points

Testing

Testing uses Playwright for browser/UI verification and the Node-RED MCP server to deploy flows and inject messages into a running Node-RED instance.

Requirements

• Node-RED >= 2.0.0
• Node.js >= 18.0.0

License

MIT

Contributing

Issues and pull requests welcome at the project repository.