@millstream/kafka
v0.5.0
Published
Apache Kafka Publisher/Subscriber for @millstream (on @platformatic/kafka): commit on ack, in-place redelivery on nack, per-partition ordering
Maintainers
Readme
@millstream/kafka
Apache Kafka Publisher / Subscriber for millstream, built on
@platformatic/kafka (pure TypeScript, no native addon). Peer dependency:
@platformatic/kafka@^1.30.
import { KafkaPublisher, KafkaSubscriber } from '@millstream/kafka';
const publisher = new KafkaPublisher({
bootstrapBrokers: ['localhost:9092'],
partitionKey: (topic, msg) => msg.metadata.order_id, // same key → same partition → ordered
});
router.addConsumerHandler({
name: 'billing',
subscribeTopic: 'orders.created',
subscriber: new KafkaSubscriber({
bootstrapBrokers: ['localhost:9092'],
consumerGroup: 'billing',
partitionConcurrency: 4,
}),
concurrency: 4,
handler: async (msg) => {
/* … */
},
});Semantics
- Ack → commit
offset + 1of that partition. Unanswered messages are never committed; after a restart or rebalance they come back (at-least-once — dedupe bymsg.uuid, e.g. withinbox()from@millstream/pg). - Nack → in-place redelivery of a fresh copy after
nackResendMs(Kafka has no per-message nack). A poison message therefore blocks its partition only: addpoisonQueue/retrymiddlewares on the Router. - Ordering: strict per partition.
partitionConcurrency: nprocesses up tonpartitions in parallel, each still sequential; give the Router handler the sameconcurrency. - New consumer groups start at the earliest offset (
fallbackMode: 'latest'to change). - uuid & metadata travel as headers (
_millstream_message_uuid+ one header per metadata key). To exchange messages with Go Watermill services, setuuidHeader: WATERMILL_UUID_HEADERon both the publisher and the subscriber. Records produced by other clients get a deterministic uuidkafka:<topic>:<partition>:<offset>. kafkaInfo(msg)returns topic / partition / offset / key / timestamp of a consumed message.
Producer notes
- The publisher's producer is idempotent by default. When part of a multi-partition Produce request fails (for
example
NOT_LEADER_OR_FOLLOWERright after topic creation or during a leader change),@platformatic/kafkaresends the whole request, including partitions the broker already acknowledged. Without idempotence those records are appended again, whilesend()still succeeds. With idempotence the broker discards the resends. - Sends are serialised per producer: with several idempotent sends in flight, one that overtakes another's retry
can land out of order (platformatic/kafka#472). For throughput,
pass many messages to one
publish()call. - Requires
@platformatic/kafka≥ 2.3: older versions silently dropped concurrent idempotent sends (#326).
