Redis Recovery

This sub-repo contains tools for recovering the state of the destination sender redis servers from data stored in the kafka clearing topics

Requirements

This application requires node.js version 11 or 12, plus the Yarn package manager.

Installation

# clone the repo
git clone [email protected]:bitbrew/hub-tools
# install packages
cd hub-tools/redis-recovery
yarn

Or install with npm or yarn

# Using npm
npm i -g bbredis-recovery
# Using yarn
yarn global add bbredis-recovery

Usage

# Working within the repository
./index.js

# Working with the globally installed tool
redis-recovery

All examples will use 'redis-recovery' rather than the local js file.

Getting help

redis-recovery --help

Configure Kafka Settings

redis-recovery kafka-config 127.0.0.1 9092 test-topic 50

Configure Redis Settings

redis-recovery redis-config 127.0.0.1 6379 Keen

Display Current Config

redis-recovery config

Export Recovery Data From Kafka

redis-recovery kafka-export

This may take a number of seconds or minutes to complete. When completed the output to the console will look similar to:

Using the following config:
Kafka topic:       test-topic
Kafka partitons:   50
Kafka host:        127.0.0.1
Kafka port:        9092
Kafka log level:   2
Redis host:        127.0.0.1
Redis port:        6379
Redis destination: Keen

✔ Initializing Kafka consumer
✔ Getting retry batches not followed by closed batches
✔ Processed 100106 records
✔ Retrieving count of queued detinations
✔ Processed 100106 records
✔ Writing results to output-kafka.json

Export Redis State

redis-recovery redis-export

This should take only a few seconds, following which the console output will be similar to:

Using the following config:
Kafka topic:       test-topic
Kafka partitons:   50
Kafka host:        127.0.0.1
Kafka port:        9092
Kafka log level:   2
Redis host:        127.0.0.1
Redis port:        6379
Redis destination: Keen

✔ Retrieving retry batches
✔ Writing results to output-redis-retry.json
✔ Retriving clearing state
✔ Writing results to output-redis-clearing.json

Import Redis State

redis-recovery redis-import

This will only take a few seconds to complete. The output will give a list of the destination- partition's that were updated in Redis:

Using the following config:
Kafka topic:       test-topic
Kafka partitons:   50
Kafka host:        127.0.0.1
Kafka port:        9092
Kafka log level:   2
Redis host:        127.0.0.1
Redis port:        6379
Redis destination: Keen

✔ Loading update file output-kafka.json
✔ Updating destination-partition 3d9eb6eb-3698-4f00-bfc7-a03d49b1efb7:49
✔ Updating destination-partition f07f8701-3ca5-4d4d-b896-3bccfea68d25:49