Quick Start: Real-time Geo-replication for ElasticSearch

ElasticSearch has significantly improved the way we operate scalable persistent services with its easy approach to scalability using symmetric nodes. One of the challenges we’ve found is operating ElasticSearch across geographically distant clusters for fault-tolerance and recovery.

In this quick start, we’ll show you how to implement ElasticSearch plugins to use as a medium for cross-cluster document replication and indexing. This tutorial is a proof-of-concept for global, multi data center replication of ElasticSearch indexing. Feel free to check out the full ElasticSearch real-time geo-replication GitHub repository here.

Note: this is not production ready! It is supposed to provoke inspiration and conversation while we work, test and get the bugs out. Thanks so much for checking it out!

Overall Process

1. Prerequisites: Internet connectivity and JAVA_HOME and PATH are configured with JDK version 7 or higher
2. Sign up for a PubNub account and get your unique pub/sub keys in the PubNub Developer’s Portal.
3. Download ElasticSearch PubNub Quickstart bundle:
4. Unzip into a local directory
5. Configure River or Changes plugin (see below)
6. Start ElasticSearch: ./bin/elasticsearch

Configuring the Changes Plugin

The full PubNub Changes Plugin for ElasticSearch GitHub repository can be seen here. This allows ElasticSearch index changes to be propagated via PubNub.

• edit conf/elasticsearch.yml

• YOUR-PUBLISH-KEY should match your PubNub publish key
• YOUR-SUBSCRIBE-KEY should match your PubNub subscribe key

Configuring the River Plugin

The full PubNub River Plugin for ElasticSearch GitHub repository can be seen here. This enables PubNub content to be indexed by ElasticSearch.

• start ElasticSearch
• run the following command using Curl

Debugging PubNub Communications

The scripts directory contains two scripts you can use to publish or listen to PubNub.

Prerequisites:

• Install node.js from https://nodejs.org/en/
• Install coffeescript using “npm install -g coffee-script”
• Install PubNub using “npm install -g pubnub”

Using pubnub_debug_listen.coffee:

• edit the script to contain your subscribe key
• run coffee pubnub_debug_listen.coffee CHANNEL_NAME, where CHANNEL_NAME is your PubNub channel name (as above, ‘elasticsearch_changes’)

Using pubnub_debug_publish.coffee:

• edit the script to contain your publish key
• edit the script to reflect your document content, index, key and version
• run coffee pubnub_debug_publish.coffee CHANNEL_NAME, where CHANNEL_NAME is your PubNub channel name (as above, ‘elasticsearch_changes’)

Summing Up Real-time Geo-replication with ElasticSearch

We hope you enjoyed this proof-of-concept example of ElasticSearch real-time geo-replication with PubNub. I also gave a talk on this topic at the Boston ElasticSearch Meetup a couple weeks ago, and you can see the full talk on Implementing Real-time Geo-Replication with ElasticSearch here. If you run into any issues or have any suggestions for making this more awesome, please drop us a line!