PubNub Storage & Playback Tutorial

Store realtime message streams for future retrieval and playback

Demo URL:

PubNub's Storage and Playback feature, also informally referred to as the History API, enables you to store messages as they are published, and retrieve the previously-published messages at a later time.

  • Populate chat, collaboration and machine configuration on app load.
  • Store message streams with real-time data management.
  • Retrieve historical data streams for reporting, auditing and compliance (HIPAA, SOX, Data Protection Directive, and more).
  • Replay live events with delivery of real-time data during rebroadcasting.
  • Chat room to allow users to view the past conversations, as they join a room
  • Collaborative drawing app to let users to view the previously created drawing lines
  • Temperature monitor to retrieve the sensor data from 15 minutes ago
  • Chess game to play back the players' moves
  • Automate machine configuration with command line steps

To use the add-on feature, you need to enable Storage and Playback for your key in the Admin Dashboard.

Enable Storage And Playback Add-on

You can choose how long the data to be stored at the configuration modal dialog.

Storage And Playback Add-on Configuration

In the dialog, you can set the duration of data storage. You can always come back to your admin dashboard to modify the settings later, however, the setting does not retroactively change the duration of previously stored messages. In other words, if the setting is 1 day and you change it to 30 days, the message published/stored when the setting was 1 day will still expire (get deleted from storage) after 1 day and not be extended to 30 days. The reverse is true, as well.

These code samples build off code defined in the Pub & Sub tutorial, so before proceeding, be sure you have completed the Pub & Sub tutorial first.

You can take a look at the Tic Tac Toe demo that explains how the APIs work, along with this documentation.

You can fetch the historical messages that have been published on a channel using history().

When you are developing a multi-player Tic-Tac-Toe game like the demo, each time a player plays, you are publishing the move, as you read in the last chapter, Publishing / Subscribing:

{
    player : "x",
    position : "2-2"
}

and let's say, you provide a Playback button so that the user can recollect the past moves. Since Tic-Tac-Toe is a simple game only with 9 moves maximum, let's collect the last 9 moves (or less).

pubnub.history(
    {
        channel : 'tic-tac-toe',
        count : 9
    },
    function (status, response) {
        console.log(response.messages);
    }
);

The response format is:

type HistoryItem = {
    timetoken : number | null,
    entry : any,
}

type HistoryResponse = {
    messages : Array<HistoryItem>,
    startTimeToken : number,
    endTimeToken : number,
}

And the response.messages outputs an array of the 9 most recently sent data (player data, for instance) on the given channel, tic-tac-toe. If the total moves for a game is less that 9, it just retrieves all data.

[
    {
        player : "x",
        position : "2-2"
    },
    {
        player : "o",
        position : "2-3"
    },
    …(7 more objects contains data)
]

By default, the last 100 messages are returned if you don't specify a count.

100 is the maximum number of messages you can fetch with the history() API. If you are creating more complex game like chess, or any other realtime applications, and wish to retrieve more than 100 messages, please refer to the section: Retrieving more than 100 messages from history.

Reverse the Traversal Order

Setting the reverse attribute to true will return messages in first-in-first-out (FIFO) order.

pubnub.history(
    {
        channel: 'tic-tac-toe',
        reverse: true
    },
    function (status, response) {
        console.log(response.messages);
    }
);

The default value is false.

Examples

Let's see how the reverse works from the simpler Tic-Tac-Toe demo.

For example, when a game ends like this.

storage and playback guide 3

and if you want to retrieve the last two moves (Move #8 and #9), this is how you do it:

pubnub.history(
    {
        channel: 'tic-tac-toe',
        count: 2
    },
    function (status, response) {
        ...
    }
);

and if you want the first two moves (Move #1 and #2), you reverse the traversal order:

pubnub.history(
    {
        channel: 'tic-tac-toe',
        reverse: true,
        count: 2
    },
    function (status, response) {
        ...
    }
);
storage and playback guide 4

The arrow shows the order of the array returned as the results.

Try this yourself using the pull-down menu and buttons within the History API Explained section of the demo to see how reversing the traversal order works.

Displaying the messages from a certain time span

To retrieve messages within certain time tokens, use start and end arguments with history():

pubnub.history(
    {
        channel: 'tic-tac-toe',
        start    : 14219808508935165,
        end      : 14219808515130421
    },
    function (status, response) {
        console.log(response)
    }
);

The time tokens must be in valid 17 digit tokens. Instead of defining the span of time with both values, you can also use either start or end, with specific count.

pubnub.history(
    {
        channel : 'tic-tac-toe',
        count : 3,
        start : 14419808508930112
    },
    function (status, response) {
        console.log(response)
    }
);

Please note that the start time is exclusive, so if you want to include the message that had been sent on a certain timestamp, you need to specify the time after the timestamp.

storage and playback guide 5storage and playback guide 6

On the other hand, the end time is inclusive.

storage and playback guide 7

When you specify both start and end, all the messages sent in the time range are returned and the count value is ignored.

storage and playback guide 8

To convert date objects to the specific time tokens, get dateObj.getTime() and multiply by 10000. For example, if you would like to retrieve the messages from 5 minutes ago from now, the value of the start parameter should be:

var now = Date.now();
var fiveMinAgo = (now - (5*60*1000)) * 10000;

and now is the value of the end parameter.

In the History API Explained section of Tic-Tac-Toe demo, click "Specific Time Span" then choose a time value from the pull down menu.

You should be able to retrieve all messages within the time span.

Paging the Messages

You can "page" the data by moving the end timestamp from the previously retrieved data as a start time to retrieve next data.

For example, when you are developing a more complex chess game, which potentially have more than 100 moves.

But before fetching 100 moves, let's take a look at this simple example, which lets you to retrieve only last two chess moves to understand how this works.

pubnub.history(
    {
        channel : 'chess',
        count : 2
    },
    function (status, response) {
        console.log(response)
    }
);

The response, m, returns with start and end timetokens:

[
    [
        move_23,
        move_24
    ],
    14401991341950176,
    14401991354356533
]

Now, use the start attribute to start the timetoken value and request next 2 data points. Use the timestamp slightly older than the time returns for move_23, to fetch move_21 and move_22.

pubnub.history(
    {
        channel : 'chess',
        count : 2,
        start : 14401991341950175
    },
    function (status, response) {
        console.log(response)
    }
);

This should return:

[
    [
        move_21,
        move_22
    ],
    14401991315643467,
    14401991328984348
]

At the end of history, the start timetoken that is returned is 0.

Paging with the reversed order gives you desired data starting from newest, in FIFO order.

Retrieving more than 100 messages from history

By default, history returns maximum 100 messages. To retrieve more than 100, repeat paging through the history, 100 at a time until you get everything.

One way to achieve this operation is:

getAllMessages = function(timetoken) {
    pubnub.history(
        {
            channel : channel,
            start : timetoken
        },
        function (status, response) {
            var msgs = response.messages;
            var start = response.startTimeToken;
            var end = response.endTimeToken;

            if (msgs.length == 100)
                getAllMessages(start);
        }
    );
}

Try PubNub Today