Build

Quick Start: Real-time Geo-replication for ElasticSearch

2 min readPubNub Staff on Nov 24, 2014
Try PubNub Today

Free up to 1MM monthly messages. No credit card required.

Subscribe to our newsletter

By submitting this form, you are agreeing to our Terms and Conditions and Privacy Policy.

This quick start shows you how to implement ElasticSearch plugins to use as a medium for cross-cluster document real-time geo-replication and indexing.

ElasticSearch has significantly improved the way we operate scalable persistent services with its easy approach to scalability using symmetric nodes. One of the challenges we’ve found is operating ElasticSearch across geographically distant clusters for fault-tolerance and recovery.

In this quick start, we’ll show you how to implement ElasticSearch plugins to use as a medium for cross-cluster document replication and indexing. This tutorial is a proof-of-concept for global, multi data center replication of ElasticSearch indexing. Feel free to check out the full ElasticSearch real-time geo-replication GitHub repository here.

Note: this is not production ready! It is supposed to provoke inspiration and conversation while we work, test and get the bugs out. Thanks so much for checking it out!

Overall Process

  1. Prerequisites: Internet connectivity and JAVA_HOME and PATH are configured with JDK version 7 or higher
  2. Sign up for a PubNub account and get your unique pub/sub keys in the PubNub Developer’s Portal.
  3. Download ElasticSearch PubNub Quickstart bundle:
  4. Unzip into a local directory
  5. Configure River or Changes plugin (see below)
  6. Start ElasticSearch: ./bin/elasticsearch

Configuring the Changes Plugin

The full PubNub Changes Plugin for ElasticSearch GitHub repository can be seen here. This allows ElasticSearch index changes to be propagated via PubNub.

  • edit conf/elasticsearch.yml
  • YOUR-PUBLISH-KEY should match your PubNub publish key
  • YOUR-SUBSCRIBE-KEY should match your PubNub subscribe key

Configuring the River Plugin

The full PubNub River Plugin for ElasticSearch GitHub repository can be seen here. This enables PubNub content to be indexed by ElasticSearch.

  • start ElasticSearch
  • run the following command using Curl

Debugging PubNub Communications

The scripts directory contains two scripts you can use to publish or listen to PubNub.

Prerequisites:

  • Install node.js from https://nodejs.org/en/
  • Install coffeescript using “npm install -g coffee-script”
  • Install PubNub using “npm install -g pubnub”

Using pubnub_debug_listen.coffee:

  • edit the script to contain your subscribe key
  • run coffee pubnub_debug_listen.coffee CHANNEL_NAME, where CHANNEL_NAME is your PubNub channel name (as above, ‘elasticsearch_changes’)

Using pubnub_debug_publish.coffee:

  • edit the script to contain your publish key
  • edit the script to reflect your document content, index, key and version
  • run coffee pubnub_debug_publish.coffee CHANNEL_NAME, where CHANNEL_NAME is your PubNub channel name (as above, ‘elasticsearch_changes’)

Summing Up Real-time Geo-replication with ElasticSearch

We hope you enjoyed this proof-of-concept example of ElasticSearch real-time geo-replication with PubNub. I also gave a talk on this topic at the Boston ElasticSearch Meetup a couple weeks ago, and you can see the full talk on Implementing Real-time Geo-Replication with ElasticSearch here. If you run into any issues or have any suggestions for making this more awesome, please drop us a line!

More from PubNub

Can Empathy Exist in the Metaverse
News May 16, 20221 min read

Can Empathy Exist in the Metaverse

A roundtable discussion led by PubNub’s COO, Casey Clegg, exploring the topics of what it means to be human in a virtual world.

PubNub Staff

PubNub Staff

How to Advance Telehealth and Virtual Care Technologies
News May 2, 20221 min read

How to Advance Telehealth and Virtual Care Technologies

Dr. Joe Kvedar, Chair of the Board for the American Telemedicine Association, joins our COO, Casey Clegg, to discuss why...

PubNub Staff

PubNub Staff

Another Step Towards Data Security: ISO-27001 Implementation
BuildMay 2, 20221 min read

Another Step Towards Data Security: ISO-27001 Implementation

Today, we are glad to announce that we are currently in the process of implementing ISO-27001 security standards.

PubNub Staff

PubNub Staff