Store, Retrieve, and Playback Data with Storage and Playback

PubNub Storage & Playback, our history API, enables you to store, retrieve, and playback messages as they were streamed. A couple use cases include:

• When a device goes offline, you need to retrieve missed messages and publish them back to the device.
• Chat applications can retrieve chat messages that were missed while a user was offline.
• Event logging
• Lat/long coordinates and drawing routes on maps

In this month’s FAQ with PubNub founder Stephen Blum and I, we’re talking all things Storage & Playback. Under the video, you’ll see a list of topics we cover in the video. Each topic covered is linked to a more robust explanation, so we hope you find this useful!

How long can messages be stored?

When Storage & Playback is enabled, you can set the duration the data is stored. Our default is one day, but it can be customized for different increments up to 30 days. However, we do have an option for forever storage as well, giving you unlimited access to any message ever published on your API keys. You can also customize what messages are stored using the History API down to the individual message.

It’s important to store messages forever in some cases for business, logistical, or legal reasons (think HIPAA), to keep your messages indexed and around forever.

How are messages stored and retrieved?

With the history API, with so many messages being stored in the network, we offer you the option to index by channel. Channels are where messages are being published, and we index by a time token (to the nearest 10 nanoseconds). You can’t request messages based on their content, and there is no full text indexing or searches. However, you can do this on the client-side if you want to.

Once I’ve retrieved messages, what’s that payload look like (aka how do I parse it?)

Everything we speak at PubNub is JSON. With the history response, it’s an array with 3 elements. The first element is an array within an array, which are your messages. The second element is a start timetoken of the first message in the message payload. And the last element of the array is the last timetoken.

So I have messages, and two timetokens, but what if I want to specify timetokens? I want all the timetokens of all the messages I’ve received. Can I do that?

With a special parameter include_token, it’ll make a change in the response payload for PubNub, and deliver a timetoken associated with every message in the response.

But does that change the way that payload looks?

Yes, it’s very different. It’s now a dictionary for every single message that you would expect to retrieve with two elements in the dictionary. The first element is message, which is your raw message, and the second element is a time token (the one you asked for), and that’s associated with the message.

How many messages can I get back at a time?

Say you have a channel with 1000 messages published to it, you’ll get 100 of the most recently published messages, starting with the most recent published and moving back from there.

But say you want the first 100 messages published, not the most recent, you’d set the reverse parameter. Messages are always returned in chronological order, so the array you get back is always going to be newest to oldest messages retrieved.

Can I retrieve less than 100 messages at a time?

Yes! To limit the number of messages retrieved at a time in a payload, we can specify one of two parameters, limit or count. Check out the documentation for which one to use (it varies based on language), but our API supports both. The maximum is always 100.

What about using time as a parameter? (7:10)

Let’s say you want the last five minutes of messages that are published. This one is a little trickier because it’s slightly more involved, but it’s totally possible. You’ll need to create your own timetoken by specifying a new date with a new date object, and then convert that to a time stamp.

So if you want 5 minutes ago, you’ll do date, current time – 5 minutes, convert that to a time stamp, then convert that time stamp into a PubNub timetoken, by multiplying it by 10,000,000. And you can then use that as a PubNub timetoken to pass through.

Can I retrieve every message I’ve ever published? (9:23)

If you want every message that’s every been published to a channel, you’ll have to page. We have a great tutorial on paging through restored PubNub messages here.

PubNub Support

Got some questions of your own? We’re here for you! Reach out to help@pubnub.com, visit PubNub Support  or tweet us @PubNub.

We also have PubNub Office Hours at our San Francisco HQ, every Wednesday from 2-5pm, where you can get your real-time questions answered in real time. Give us a heads up at office-hours@pubnub.com and let us know you’re coming.