Build

Get Redis Replication Right: Multi Data Center Support

3 min read PubNub Staff on Aug 26, 2014
Try PubNub Today

Free up to 1MM monthly messages. No credit card required.

Subscribe to our newsletter

By submitting this form, you are agreeing to our Terms and Conditions and Privacy Policy.

This tutorial walks you through a proof-of-concept for global, multi data center Redis replication of data using PubNub and NodeJS. We're continuing to grow our collection of persistence adapters for ...

Redis Replication

This tutorial walks you through a proof-of-concept for global, multi data center Redis replication of data using PubNub and NodeJS. We’re continuing to grow our collection of persistence adapters for PubNub using Node.JS.

Redis replication can be difficult due to TCP windowing and other Kernal tuning obstacles. In this blog post, we’ll show you one way to approach a replication model, specifically intended for low throughput and high volume. And with PubNub, you’ll get multi data center Redis replication, a globally redundent method for replicating Redis data.

Note: this is not production ready! It is supposed to provoke inspiration and conversation while we work, test and get the bugs out. Thanks so much for checking it out!

Before You Begin with Redis Replication

  • You’ll first need to sign up for a PubNub account. Once you sign up, you can get your unique PubNub keys in the PubNub Developer Portal. You’ll use your own subscription key for any of these samples.
  • Note that RDB-based replication only works with RDB file format 2.4 (hopefully 2.6+ coming soon, but in the meantime updates work).
  • Note that all dependencies should be included in the “deps” folder – let us know if we missed one!
  • If you need to know the revision or source of a dependency, check the GIT_REV or .git_config files in that directory respectively.
  • Note that this proof-of-concept currently only supports a small subset of Redis write commands.
  • Note that this has not been scale tested! It is merely nifty and awesome.

Understanding the Two Phases of Redis Slave Replication

Redis replication consists of 2 phases:

  • Phase 1: the slave connects to the master and gets a dump of all entities as an RDB stream
  • Phase 2: the slave starts receiving updates in real time

The current Redis replicator can read RDB files from Redis 2.4 and earlier. We’re investigating adding support for 2.6+ soon! Even if Phase 1 isn’t working for you, it is very likely that Phase 2 will work for receiving real-time updates.

Running the Code: Starting a Redis Replicator

To start a Redis replication client, use this command:

This starts up the replication process. You should see something like this:

What happened? The replication client connected to your redis instance, started an RDB replication transfer, and sent all of the entities to the PubNub channel “redisrepl”.

The PING commands are harmless – they are just there because the replication client needs to stay connected to the server.

That wasn’t bad, was it? Let’s try an update!

You should see something like this:

Pretty awesome! That means that the replication client saw the update, and sent it out on the “redisrepl” channel.

Running the Code: Starting a Replication Listener

To start a Redis replication listener, use this command:

This starts a listener on the given PubNub channel. You can modify the listener to do whatever you want with the data it receives (this one just logs to console).

As you perform updates, you should see messages like this:

Running the Code: Starting a Replication Proxy

This starts a listener on the given PubNub channel that proxies data to another Redis instance on port 6399.

As you perform updates, you should see messages like this:

If you check the redis instance on port 6399, you should see something like this:

Summing Up Redis Replication

We hope you enjoyed this example of replicating Redis data using PubNub and Node.JS. In the future, we’ll implement and review more Redis commands, and start getting this sample code a bit more production ready. If you run into any issues or have any suggestions for making this more awesome, please drop us a line!

More from PubNub

Top Trends to Engage Your App Users
InsightsSep 27, 20224 min read

Top Trends to Engage Your App Users

Take a look at the top trends that are the most effective in attracting customers and reducing churn, and how you can incorporate...

PubNub Staff

PubNub Staff

Comparing Game Engines: Unity vs Unreal vs the Rest
Real-Time Gaming BlogSep 21, 20225 min read

Comparing Game Engines: Unity vs Unreal vs the Rest

Comparing the major game engines: Unity vs Unreal Engine vs Corona SDK vs GameMaker Studio, including the benefits and cons of...

PubNub Staff

PubNub Staff

Python Socket Programming: Client, Server and Peer Libraries
BuildSep 21, 20226 min read

Python Socket Programming: Client, Server and Peer Libraries

Sockets (aka socket programming) enable programs to send and receive data, bi-directionally, at any given moment. This tutorial...

PubNub Staff

PubNub Staff