Real-Time Gaming Blog

A/B Testing Your Game with PubNub and Optimizely

3 min readJordan Schuetz on Aug 16, 2018
Try PubNub Today

Free up to 1MM monthly messages. No credit card required.

Subscribe to our newsletter

By submitting this form, you are agreeing to our Terms and Conditions and Privacy Policy.

How to run A/B testing for your multiplayer game to identify how players are performing on different types of levels.

A/B testing is an important tool for modern game designers and product managers.  Conducting A/B tests provide a robust framework to facilitate experimentation with many aspects of the game in a controlled way and to confidently make decisions about positive changes to the game’s design.  There are many services that offer A/B Testing like Optimizely, Google Analytics, and open source.

In today’s tutorial, we are going to take a look at how easy it was to integrate Optimizely A/B testing into our Multiplayer Ninja Platformer PubNub game.

Integration

The goal of this experiment was to see which level in the multiplayer game provided the highest amount of engagement.  Additionally, we wanted to test how easy each level was to complete by taking the percentage of people who actually went through the door to move on to the next stage.  In order to measure this information, we need to setup three different variation variables in the Optimizely portal, and an additional completedLevel metric which tracks when the player enters the door and changes scenes.

Integration

Below are some examples of what or A/B test looks like in the Optimizely portal.  Each variation is divided into even increments of 33.3% where accuracy improves as more users visit the webpage.

A/B test

The first step to implement the Optimizely SDK is to simply require it in your index.html document:

<script type="text/javascript" src="https://cdn.optimizely.com/optimizely-2.1.2.min.js"></script>

Next, grab your Optimizely ID from the portal, and place it as the optimizelyID variable.  The following request grabs your Optimizely data file which downloads your settings and A/B testing variables.

var optimizelyID = 'YOUR-ID-HERE' // ID of full stack project
// Function to request JSON datafile to initalize Optimizely SDK
var getDatafile = () => {
  return new Promise( (resolve, reject) => {
    var request = new XMLHttpRequest();
    request.open('GET', `https://cdn.optimizely.com/json/${optimizelyID}.json`);
    request.onload = () => {
      if (request.status >= 200 && request.status < 400) {
        var resp = request.responseText;
        resolve(resp);
      } else {
        reject('Unable to capture datafile');
      }
    };
    request.onerror = function() {
      reject('Unable to capture datafile');
    };
    request.send();
  });
}

Once you have received the data file, you can initialize the Optimizely A/B test.  In this code, we check to see if Optimizely is enabled, make the current level either 0,1,2 depending upon the A/B testing probabilities.

// Initializing Optimizely SDK & resetting default level based on bucket
getDatafile().then((datafile) => {
  // Initalize the client after getting the datafile
  window.optlyClient = window.optimizelyClient.createInstance({datafile: datafile});
  // Bucket user and send impression to Optimizely
  var enabled = window.optlyClient.isFeatureEnabled('ninjaMultiplayer', window.UniqueID);
  // set a new level to start at 
  window.startingLevel = window.optlyClient.getFeatureVariableInteger('ninjaMultiplayer', 'startingLevel', window.UniqueID);
  if(enabled){
    window.globalCurrentLevel = startingLevel;
  };
});

Additionally, we can track when a user completes a level by inserting this code when the player enters the door, which runs the _goToNextLevel() function.

window.optlyClient.track('completedLevel', window.UniqueID);

Once the code is implemented and pushed live, you can simply change user interactions through the Optimizely portal.

Conclusion

Optimizely is a well integrated solution for A/B testing that implements well with PubNub’s Real-time API’s.  If you want to learn more about how you can use PubNub to create this real-time game yourself, check out this tutorial.

More from PubNub

Can Empathy Exist in the Metaverse
News May 16, 20221 min read

Can Empathy Exist in the Metaverse

A roundtable discussion led by PubNub’s COO, Casey Clegg, exploring the topics of what it means to be human in a virtual world.

PubNub Staff

PubNub Staff

How to Advance Telehealth and Virtual Care Technologies
News May 2, 20221 min read

How to Advance Telehealth and Virtual Care Technologies

Dr. Joe Kvedar, Chair of the Board for the American Telemedicine Association, joins our COO, Casey Clegg, to discuss why...

PubNub Staff

PubNub Staff

Another Step Towards Data Security: ISO-27001 Implementation
BuildMay 2, 20221 min read

Another Step Towards Data Security: ISO-27001 Implementation

Today, we are glad to announce that we are currently in the process of implementing ISO-27001 security standards.

PubNub Staff

PubNub Staff