Multiple Data Center Redis Replication with NodeJS

This tutorial walks you through a proof-of-concept for global, multi data center Redis replication of data using PubNub and NodeJS. We’re continuing to grow our collection of persistence adapters for PubNub using Node.JS.

Redis replication can be difficult due to TCP windowing and other Kernal tuning obstacles. In this blog post, we’ll show you one way to approach a replication model, specifically intended for low throughput and high volume. And with PubNub, you’ll get multi data center Redis replication, a globally redundent method for replicating Redis data.

Note: this is not production ready! It is supposed to provoke inspiration and conversation while we work, test and get the bugs out. Thanks so much for checking it out!

Before You Begin with Redis Replication

You’ll first need to sign up for a PubNub account. Once you sign up, you can get your unique PubNub keys in the PubNub Developer Portal. You’ll use your own subscription key for any of these samples.
Note that RDB-based replication only works with RDB file format 2.4 (hopefully 2.6+ coming soon, but in the meantime updates work).
Note that all dependencies should be included in the “deps” folder – let us know if we missed one!
If you need to know the revision or source of a dependency, check the GIT_REV or .git_config files in that directory respectively.
Note that this proof-of-concept currently only supports a small subset of Redis write commands.
Note that this has not been scale tested! It is merely nifty and awesome.

Understanding the Two Phases of Redis Slave Replication

Redis replication consists of 2 phases:

Phase 1: the slave connects to the master and gets a dump of all entities as an RDB stream
Phase 2: the slave starts receiving updates in real time

The current Redis replicator can read RDB files from Redis 2.4 and earlier. We’re investigating adding support for 2.6+ soon! Even if Phase 1 isn’t working for you, it is very likely that Phase 2 will work for receiving real-time updates.

Running the Code: Starting a Redis Replicator

To start a Redis replication client, use this command:

This starts up the replication process. You should see something like this:

What happened? The replication client connected to your redis instance, started an RDB replication transfer, and sent all of the entities to the PubNub channel “redisrepl”.

The PING commands are harmless – they are just there because the replication client needs to stay connected to the server.

That wasn’t bad, was it? Let’s try an update!

You should see something like this:

Pretty awesome! That means that the replication client saw the update, and sent it out on the “redisrepl” channel.

Running the Code: Starting a Replication Listener

To start a Redis replication listener, use this command:

This starts a listener on the given PubNub channel. You can modify the listener to do whatever you want with the data it receives (this one just logs to console).

As you perform updates, you should see messages like this:

Running the Code: Starting a Replication Proxy

This starts a listener on the given PubNub channel that proxies data to another Redis instance on port 6399.

As you perform updates, you should see messages like this:

If you check the redis instance on port 6399, you should see something like this:

Summing Up Redis Replication