Multiple Data Center MongoDB Replication

Welcome to a growing collection of persistence adapters for PubNub using Node.JS. In this sample, we present a proof-of-concept for global, multi data center MongoDB replication data using PubNub.

Note: this is not production-ready! It is supposed to provoke inspiration and conversation while we work, test and get the bugs out. Thanks so much for checking it out!

Before You Begin MongoDB Replication

• You’ll first need to sign up for a PubNub account. Once you sign up, you can get your unique PubNub keys in the PubNub Developer Portal. You’ll use your own subscription key for any of these samples, so once you have your keys, clone the GitHub repo and enter your unique keys on the initialization.
• Download & install mongodb version 2.4.10 – let us know if you’re using a different version you’d like us to check out!
• Install the node.js mongodb native driver using npm install mongodb -g
• IMPORTANT! Make sure that you register for a new PubNub account, and use your own subscription key for any of these samples.
• Note that all dependencies should be included in the “deps” folder – let us know if we missed one!
• If you need to know the revision or source of a dependency, check the GIT_REV or .git_config files in that directory respectively
• Note that this has not been scale tested! It is merely nifty and awesome…

MongoDB Configuration

For this example, try out the following steps when configuring a test MongoDB instance:

1. Start MongoDB using a replication set argument, for example: $ ./bin/mongod --replSet test --dbpath ./data
2. Start a mongo shell and configure MongoDB as follows: > rs.initiate({_id: "test", members: [{_id: 0, host: "127.0.0.1:27017"}]})

Understanding MongoDB Replication

MongoDB uses Replication Sets as a primary grouping mechanism for replication. Within each replica set, one node is configured as a primary. If the cluster detects a primary node failure, a secondary node will be elected to become a new master. The oplog is replicated to each node and contains itemized updates which are applied to each node’s state in-order to determine the current state of the collection. The oplog also has a fixed size which determines the number of events stored in its history.

For more information, check out:

• MongoDB Replication
• More about the MongoDB Replication process
• The MongoDB Oplog

Running the code: Starting a MongoDB Replicator

To start a MongoDB replication client, use this command:

This starts up the replication process. You should see something like this:

Now, perform a few operations on your collection, for example:

• insert {"email":"dude@dude.com"}
• insert {"email":"dude@dude.com"} (again)
• insert {"email":"dude@dude.com"} (again)
• delete {"email":"dude@dude.com"}

If you’re using the mongodb nodejs client, you’d do something like this:

Whoa, that’s pretty awesome! You should see something like:

Pretty awesome! That means that the replication client saw the update, and sent it out on the “mongochan” channel.

What happened? The replication client connected to your MongoDB instance, started listening to the oplog, and sent oplog change events to the PubNub channel “mongochan”. The cool thing is that even though it was one operation, the “delete” operation removed three records under the hood, so there are three entries in the oplog.

Running the code: Starting a replication listener

To start a MongoDB replication listener, use this command:

This starts a listener on the given pubnub channel. You can modify the listener to do whatever you want with the data it receives! (This one just logs to console)

As you perform updates, you should see messages like this:

Summing Up MongoDB Replication

We hope you enjoyed this example of replicating MongoDB data using PubNub and Node.JS. In the future, we’ll implement and review more MongoDB functionality, and start getting this sample code a bit more production-ready. If you run into any issues or have any suggestions for making this more awesome, please drop us a line!