Feedback

Node.jsNode.jsPhoneGapReactWebJavaScriptNode.JS V4 Storage & Playback Tutorial for Realtime Apps

 

These docs are for PubNub 4.0 for JavaScript which is our latest and greatest! For the docs of the older versions of the SDK, please check PubNub 3.0 for JavaScript, PubNub 3.0 for NodeJS and PubNub 3.0 for PhoneGap.

If you have questions about the PubNub for JavaScript SDK, please contact us at support@pubnub.com.

Requires that the Storage and Playback add-on is enabled for your key. How do I enable add-on features for my keys? - see http://www.pubnub.com/knowledge-base/discussion/644/how-do-i-enable-add-on-features-for-my-keys
PubNub's Storage and Playback feature enables developers to store messages as they are published, and retrieve them at a later time. Before using this feature it must be enabled in the PubNub Admin Console.
Being able to pull an archive of messages from storage has many applications:
  • Populate chat, collaboration and machine configuration on app load.
  • Store message streams with real-time data management.
  • Retrieve historical data streams for reporting, auditing and compliance (HIPAA, SOX, Data Protection Directive, and more).
  • Replay live events with delivery of real-time data during rebroadcasting.
As needed, specific messages can be marked "do not archive" when published to prevent archiving on a per-message basis, and storage retention time can range from 1 day to forever.
These code samples build off code defined in the Pub & Sub tutorial, so before proceeding, be sure you have completed the Pub & Sub tutorial first.
To begin, lets populate a new channel with some test publishes that we'll pull from storage using the history() method.
function pub() {
    for (var i = 0; i < 500; i++) {
        // publish 500 messages...
        pubnub.publish({
            channel : 'history_channel',
            message : "message : " + i
        });       
    }
}

var pubnub = new PubNub({ 
    /* initiation arguments */
})
pubnub.addListener({
    status: function(statusEvent) {
        if (statusEvent.category === "PNConnectedCategory") {
            pub();
        }
    },
    message: function(message) {
        // handle message
    },
    presence: function(presenceEvent) {
        // handle presence
    }
})
 
pubnub.subscribe({ 
    channels: ['history_channel'] 
});
In the above example, we subscribe to history_channel, and onConnect, we'll publish a barrage of test messages to it. You should see these messages as they are published on the console.
Now that we've populated storage, we can pull from storage using the history() method call:
pubnub.history(
    {
        channel: 'history_channel',
        reverse: true, // Setting to true will traverse the time line in reverse starting with the oldest message first.
        count: 100, // how many items to fetch
        stringifiedTimeToken: true, // false is the default
        start: '123123123123', // start time token to fetch
        end: '123123123133' // end timetoken to fetch
    },
    function (status, response) {
        // handle status, response
    }
);
The response format is
[
	["message1", "message2", "message3",... ],
	"Start Time Token",
	"End Time Token"
]
By default, history() returns the last 100 messages, first-in-first-out (FIFO). Although we specified 100 for the count, that is the default, and if we hadn't specified a count, we would have still received 100. Try this example again, but specify 5 for count to see what is returned.
Setting the reverse attribute to true will return the first 100 messages, first-in-first-out (FIFO). Try this example again, but specify true for reverse to see what is returned.
Given this example, you will get the last two messages published to the channel:
pubnub.history(
    {
        channel: 'history_channel',
        reverse: false, // true to send via post
        count: 2 // how many items to fetch
    },
    function (status, response) {
        // handle status, response
    }
);
[[498,499],14272454823679518,14272454845442942]
To page for the next 2, use the set the start attribute to start timetoken value, and request again with all other settings the same:
[[496,497],14272454791882580,14272454823679518]
As illustrated, paging with the default reverse as true pages 2 at-a-time starting from newest, in FIFO order. If you repeat these Paging example steps with reverse as false, you will page 2 at-a-time as well, starting instead from the oldest messages, but still in FIFO order. You will know you are at the end of history when the returned start timetoken is 0.
To pull from a slice of time, just include both start and end attributes:
pubnub.history(
    {
        channel: 'history_channel',
        count: 100, // how many items to fetch
        start: 13827485876355504, // start time token to fetch
        end: 13827475876355504 // end timetoken to fetch
    },
    function (status, response) {
        // handle status, response
    }
);
The timetoken response value is a string, representing 17-digit precision unix time (UTC). To convert PubNub's timetoken to Unix timestamp (seconds), divide the timetoken number by 10,000,000 (10^7).
// init PubNub
var pubnub = new PubNub({
    publishKey   : 'pub-...',
    subscribeKey : 'sub-...'
})

We want to create a channel with messages that we can easily understand the order of those message: message #1, message #2, etc.

We use setInterval to add some delay between the published messages. First, we want to be sure that the messages are stored in the order we publish (remember, it's all async, including on the server side) and we want some time spacing between our message timetokens.

function pub(channel, total) {
    var i = 1;
 
    var looper = setInterval(
        function() {
            pubnub.publish({
                channel: channel,
                message: "message #" + i++
            });
 
            if (i > total) {
                clearInterval(looper);
            }
        }, 
    400);
}

You can create a several different test channels with different number of messages but one with 32 message ought to do the trick for simple history paging testing. You want a non-round number so you can page by 5 or 10 and see what the uneven end results gives you. It is useful to have another channel with round number to see that as well.

It's a bit crude but it gets the job done without too much fanciness to get in the way of pointing out what is happening. It uses recursion and not optimally.

function getMessages(args, callback) {
    pubnub.history(
        {
            // search starting from this timetoken
            start: args.startToken,
            channel: args.channel,
            // false - search forwards through the timeline
            // true - search backwards through the timeline
            reverse: args.reverse,
            // limit number of messages per request to this value; default/max=100
            count: args.pagesize,
            // include each returned message's publish timetoken
            includeTimetoken: true,
            // prevents JS from truncating 17 digit timetokens
            stringifiedTimeToken: true
        },
        function(status, response) {
            // holds the accumulation of resulting messages across all iterations
            var results = args.results;
            // the retrieved messages from history for this iteration only
            var msgs = response.messages;
            // timetoken of the first message in response
            var firstTT = response.startTimeToken;
            // timetoken of the last message in response
            var lastTT = response.endTimeToken;
            // if no max results specified, default to 500
            args.max = !args.max ? 500 : args.max;
 
            if (msgs != undefined && msgs.length > 0) {
                // display each of the returned messages in browser console
                for (var i in msgs) {
                    msg = msgs[i];
                    console.log(msg.entry, msg.timetoken);
                }
 
                // first iteration, results is undefined, so initialize with first history results
                if (!results) results = msgs;
                // subsequent iterations, results has previous iterartions' results, so concat
                // but concat to end of results if reverse true, otherwise prepend to begining of results
                else args.reverse ? results = results.concat(msgs) : results = msgs.concat(results);
            }
 
            // show the total messages returned out of the max requested
            console.log('total    : ' + results.length + '/' + args.max);
 
            // we keep asking for more messages if # messages returned by last request is the
            // same at the pagesize AND we still have reached the total number of messages requested
            // same as the opposit of !(msgs.length < pagesize || total == max)
            if (msgs.length == args.pagesize && results.length < args.max) {
                getMessages(
                    {
                        channel:args.channel, max:args.max, reverse:args.reverse, 
                        pagesize:args.pagesize, startToken:args.reverse ? lastTT : firstTT, results:results
                    }, 
                    callback);
            }
            // we've reached the end of possible messages to retrieve or hit the 'max' we asked for
            // so invoke the callback to the original caller of getMessages providing the total message results
            else callback(results);
        }
    );
}

We will assume that we have a channel named test1 with 32 messages in Storage as follows:

  • message #1
  • message #2
  • ...
  • message #32

each with a unique timetoken.

Invoke the getMessages function to get the following desired results:

  1. getMessages(
        {
            channel: 'test1',
            max: 50,
            pagesize: 5,
            reverse: true
        },
        function(results) {
            console.log("results: \n" + JSON.stringify(results));       
        }
    );
  2. getMessages(
        {
            channel: 'test1', 
            max: 20, 
            pagesize: 5,
            reverse: true
        },
        function(results) {
            console.log("results: \n" + JSON.stringify(results));       
        }
    );
  3. getMessages(
        {
            channel: 'test1',
            max: 50,
            pagesize: 5,
            reverse: false
        },
        function(results) {
            console.log("results: \n" + JSON.stringify(results));       
        }
    );
  4. getMessages(
        {
            channel: 'test1',
            max: 20,
            pagesize: 5,
            reverse: false
        },
        function(results) {
            console.log("results: \n" + JSON.stringify(results));       
        }
    );

    So you should see that the order of the returned messages of each invocation of history is always oldest to newest (ascending). But the order in which the messages are searched depends on the reverse parameter.

    • reverse:true will retrieve blocks of message from the oldest to the newest: 1,2,3,4,5 | 6,7,8,9,10 | ...
    • reverse:false will retrieve blocks of message from the newest to the oldest: 28,29,30,31,32 | 23,24,25,26,27 | ...

    So to reiterate, order of the messages in each response is the same, but the direction of search/retrieve is different.

     
    This is something the new Storage/History design will clear up with simpler, more robust APIs (for example: getMessagesSince, getMessagesBefore, getMessagesBetween).

    Now let's introduce the startToken parameter so we can retrieve message from or to a given point in the channel's history timeline. The timetoken you specify will be one that you need to pick from an existing message in the channel's history. It will be different for you than for me since we published at different times with different keys. I will use startToken:14774567814936359 for the following examples and let's assume this is the timetoken for message #17.

  5. getMessages(
        {
            channel: 'test1',
            max: 20,
            pagesize: 5,
            reverse: true,
            startToken:"14774567814936359"
        }, 
        function(results) {
            console.log("results: \n" + JSON.stringify(results));       
        }
    );
     
    You have to pass the timetoken values as strings (notice that startToken param value is quoted above, and below) otherwise JavaScript will round them up. For example, 14774567814936359 will become 14774567814936360 and you will get unexpected results. Go ahead and try it without the quotes and you will see that the message at the provided timetoken will be retrieved, but it should be excluded. You can see the rounded up timetoken in the history URL that was submitted by checking in the Network tab of your browser.
  6. getMessages(
        {
            channel: 'test1',
            max: 50,
            pagesize: 5,
            reverse: false,
            startToken:"14774567814936359"
        }, 
        function(results) {
            console.log("results: \n" + JSON.stringify(results));       
        }
    );

    We could go on and on with more examples, but I'll let you (Scott) and others ask for how-to examples and I can provide as necessary.

The getMessages function assembles the resulting array of messages in a presorted ascending order even when messages are retrieve from newest to oldest order. It does this by prepending the new history results with the current accumulation of previous iterations' results.

If the desire is to sort messages in descending timetoken order, then it is fairly simple to implement a sort comparator function to do so. I provide a simple example of this here:

function sortHistory(messages, desc, callback) {
    messages.sort(function(a, b) {
        var e1 = desc ? b : a;
        var e2 = desc ? a : b;
        return parseInt(e1.timetoken) - parseInt(e2.timetoken);
    });
 
    callback(messages);
}

This sortHistory function accepts the messages array (an array of JSON elements: {message, timetokens}), a flag that indicates the desired sort order (ascending is default, set to true for descending), and a function that will be called with the resulting sorted array messages.

Now let's call getMessages with reverse:true. This means that the messages will be retrieved in oldest to newest and if we want ascending sort order, then it is already in that order.

getMessages(
    {
        channel: 'test1',
        max: 100,
        pagesize: 5,
        reverse: true
    }, 
    function(results) {
        console.log("presorted: \n" + JSON.stringify(results));       
    }
);

If we want to sort the messages in descending order, we just add a call to our sortHistory function.

getMessages(
    {
        channel: 'test1',
        max: 100,
        pagesize: 5,
        reverse: true
    }, 
    function(results) {
        console.log("before sort: \n" + JSON.stringify(results));
 
        // sort messages in descending order
        sortHistory(results, true, function(sorted) {
            console.log("after sort: \n" + JSON.stringify(sorted));
        });        
    }
);

And if we retrieve the message with reverse:false, newest to oldest, then we still get the messages assembled in ascending order even though the iterations in chunks of size pagesize (16,17,18,19,20 | 11,12,13,14,15...). But if we want to sort descending, again, just need to call the sortHistory function.

getMessages(
    {
        channel: 'test1',
        max: 100,
        pagesize: 5,
        reverse: false
    }, 
    function(results) {
        console.log("before sort: \n" + JSON.stringify(results));

        // sort messages in descending order
        sortHistory(results, true, function(sorted) {
            console.log("after sort: \n" + JSON.stringify(sorted));
        });        
    }
);