node-red-kafka

Apache Kafka producer and consumer nodes for Node-RED, built on node-red-plugincore.

What is Kafka?

Apache Kafka is a distributed event streaming platform. At its core it works like a persistent, high-throughput message bus:

• Topics - named channels that producers write to and consumers read from. Think of a topic as a log file that multiple services can append to and tail simultaneously.
• Brokers - the Kafka server(s) that store and serve messages. A cluster has one or more brokers.
• Producers - clients that publish messages to a topic.
• Consumers - clients that subscribe to a topic and process messages as they arrive.
• Consumer Groups - a group of consumers that share the workload of a topic. Each partition is assigned to exactly one consumer in the group at a time.
• Partitions - each topic is split into partitions for parallelism and scalability. A key on a message controls which partition it lands on.
• Offsets - every message in a partition has a sequential offset number. Kafka remembers where each consumer group is up to, so consumers can resume after a restart without missing or re-reading messages.

Kafka is commonly used for real-time data pipelines, event-driven architectures, IoT telemetry ingestion, and microservice integration.

Nodes

| Node | Type | Description | |------|------|-------------| | Kafka Cluster | Config | Connection details for a Kafka broker or cluster | | Kafka Topic | Config | Names a topic and links it to a cluster | | Kafka Consumer | Output | Subscribes to a topic and emits a message per record | | Kafka Producer | Input | Publishes an incoming message to a topic |

[!IMPORTANT] This plugin requires @theotherwillembotha/node-red-plugincore to be installed.

node-red-plugincore is declared as a dependency and npm will install it automatically alongside this package. However, due to a known Node-RED limitation, packages that arrive as transitive npm dependencies are only discovered by the Node-RED runtime on the next startup.

You have two options:

• Install @theotherwillembotha/node-red-plugincore via the palette manager or npm install first, then install this plugin - both will be available immediately without a restart.
• Install this plugin directly - node-red-plugincore will be installed automatically alongside it. Restart Node-RED once and both packages will be fully loaded.

Getting Started

1. Start a Kafka broker

Save the following as kafka-compose.yaml and run docker compose -f kafka-compose.yaml up -d:

services:
  kafka:
    image: apache/kafka:4.3.0
    container_name: kafka
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_NUM_PARTITIONS: 3
    networks:
      - dev_nodered

networks:
  dev_nodered:
    name: dev_nodered
    external: true

Note: KAFKA_LISTENERS must bind to 0.0.0.0 (not localhost) so that other containers on the same Docker network can reach the broker. KAFKA_ADVERTISED_LISTENERS must use the container name (kafka) so that clients resolve it correctly via Docker DNS.

The Kafka container and your Node-RED container must be on the same Docker network (or otherwise reachable from each other). The compose file above attaches Kafka to dev_nodered - make sure your Node-RED container is on that same network. If you are using a different network name, update the networks section accordingly. You can verify connectivity by pinging the kafka container from within the Node-RED container before configuring the cluster node.

2. Configure the cluster in Node-RED

Add a Kafka Cluster config node and point it at kafka:9092. Use Test Connection to verify before saving.

3. Add a topic

Add a Kafka Topic config node, select your cluster, and enter a topic name.

4. Wire up consumer and/or producer nodes

Drop a Kafka Consumer or Kafka Producer node into your flow and select the topic config.

Node Reference

Kafka Cluster Config

Connection configuration for a Kafka broker or cluster. All other Kafka nodes reference this.

| Field | Description | |-------|-------------| | Name | Display label for this config | | Client ID | Identifier sent to Kafka to identify this Node-RED instance. Defaults to node-red | | Brokers | Comma-separated list of broker addresses in host:port format, e.g. kafka:9092 or b1:9092,b2:9092,b3:9092 | | SSL | Enable SSL/TLS encryption | | SASL | Authentication mechanism - None, PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512 |

Use the Test Connection button to verify connectivity before saving. The test times out after 6 seconds.

Kafka Topic Config

Names a Kafka topic and links it to a cluster. Both consumer and producer nodes reference this, so the topic name is defined in one place.

| Field | Description | |-------|-------------| | Name | Display label for this config | | Cluster | The Kafka Cluster config node to use | | Topic | The Kafka topic name, e.g. my-events or orders.created |

Kafka Consumer

Subscribes to a Kafka topic and emits a Node-RED message for each record received.

| Field | Description | |-------|-------------| | Name | Display label | | Topic | The Kafka Topic config node to consume from | | Group ID | Consumer group identifier. Consumers in the same group share the partition load and Kafka tracks their offset independently per group | | Read from beginning | If enabled, starts from the earliest available offset when no committed offset exists for this group |

Warning - Read from beginning: This flag only takes effect the first time a consumer group connects to a topic (i.e. when there is no committed offset for that group). On a busy or long-lived topic this can replay millions of historical messages, potentially flooding your flow. Only enable this on a fresh consumer group, or when you specifically intend to reprocess historical data. To reset, either change the Group ID or manually reset offsets via a Kafka admin tool.

Output message properties:

| Property | Type | Description | |----------|------|-------------| | msg.payload | any | Message value - parsed as JSON if possible, otherwise a plain string | | msg.topic | string | The Kafka topic name | | msg.partition | number | Partition the message came from | | msg.offset | string | Message offset within the partition | | msg.key | string | Message key, if present | | msg.headers | object | Kafka message headers |

The node status shows Connected (green) when the consumer is running, or Error (red) if the connection fails.

Kafka Producer

Publishes an incoming Node-RED message to a Kafka topic.

| Field | Description | |-------|-------------| | Name | Display label | | Topic | The Kafka Topic config node to publish to | | Key Field | Optional. A value to use as the Kafka message key, resolved from msg, flow, global, or as a literal string. Leave blank to send keyless messages |

Input: msg.payload is used as the message value. Objects are serialised to JSON automatically; all other values are converted to a string.

Output: The original input message is forwarded unchanged after the publish completes successfully.

The node status shows Connected (green) when ready, or Error (red) if the connection fails. Messages arriving before the producer is connected will trigger an error.

A note on message keys: Kafka uses the key to determine which partition a message is routed to. Messages with the same key always land on the same partition, which guarantees ordering for that key. If no key is set, messages are distributed round-robin across partitions.

Example

The flow below uses an Inject node to trigger a producer, then a consumer reads it back and logs it.

Requirements

• Node-RED >= 3.0.0
• Node.js >= 18.0.0
• A running Apache Kafka broker (see Getting Started above)

License

ISC