Build

Ingest, Store + Analyze IoT Analytics in Real time w/ mnubo

9 min read • Michael Carroll on Aug 7, 2017

Mnubo’s IoT analytics platform helps product manufacturers ingest, analyze and extract valuable insights from their connected product data by providing automated reports and dashboards that help the manufacturer understand product usage, performance, failures, operating conditions and other critical KPIs.

In this tutorial, we’ll dive into a simple example of how to track and display IoT events from a real-time Angular2 web application using the mnubo SmartObjects block. For those unfamiliar with the PubNub BLOCKS Catalog, it is a collection of prebuilt functions that allow you to run serverless functions on your real-time applications.

Myapp IoT Events Integration

As we prepare to explore our sample web application, let’s dive deeper into the underlying mnubo API.

mnubo API

Large-scale IoT monitoring and data analysis is a notoriously difficult area. Designing, developing and deploying IoT analytic insights requires skills that are different than most traditional software projects. Therefore connected object manufacturers/OEMs, who want to operate as quickly as possible with a feedback loop between their data and relevant insights, should use pre-built self-service data analytics and data science services

mnubo’s turnkey solution has a friendly UI and easy-to-use API that “just works.” The API and user interface instantly deliver, at scale, out-of-the-box insights to understand operations, faults, product usage, customer engagement, churn, etc. To top it all off, mnubo’s analytic libraries seamlessly integrate with custom (external) dashboards.

mnubo API

Since you’re reading this at PubNub, we’ll assume you already have a real-time application use case in mind. In the sections below, we’ll focus on the the IoT metrics use case, saving other analytic use cases for the future.

Obtaining Your PubNub Developer Keys

You’ll first need your PubNub publish/subscribe keys. If you haven’t already, sign up for PubNub. Once you’ve done so, go to the PubNub Admin Portal to get your unique keys.

The publish and subscribe keys look like UUIDs and start with “pub-c-” and “sub-c-” prefixes respectively. Keep those handy – you’ll need to plug them in when initializing the PubNub object in your HTML5 app below.

Getting Started with mnubo

To get started with mnubo, and to take advantage of their IoT APIs, you will need to create a SmartObjects account.

  1. Start by going to the mnubo signup page
  2. Next, go to the devices panel and click the “upgrade plan” button
  3. After your ‘trial request’ is approved, you can obtain your API keys here

Overall, it’s a quick process..and free to get started, which is a bonus!

Setting up the BLOCK

With PubNub BLOCKS, it’s really easy to create code to run in the network. And all our PubNub BLOCKS catalog items run on Functions. Here’s how to make it happen:

Go to the application instance on the PubNub Admin Dashboard.

PubNub BLOCKS catalogCreate a new Function.

Create a new FunctionPaste in the Function code from the next section.

Start the Function, and test it using the “publish message” button and payload on the left-hand side of the screen (including the Auth Key from the steps above).

Start the Function

That’s all it takes to create your serverless code running in the cloud!

Diving into the Code – the BLOCK

You’ll want to grab the 100 lines of BLOCK JavaScript and save them to a file, say, pubnub_mnubo_function.js. It’s available as a Gist on GitHub for your convenience.

First up, we declare our dependency on xhr (for HTTP requests), kvstore (to store API tokens), and basicAuth (to create Basic Auth headers). We’ll use all these to create a function to handle incoming messages from PubNub.

const XHR = require('xhr');
const db = require('kvstore');
const basicAuth = require('codec/auth');

This next function is some utility code for obtaining a temporary API token from the mnubo API via XHR.

// See mnubo API
function fetchAccessToken(authUri, clientId, clientSecret) {
    const auth = basicAuth.basic(clientId, clientSecret);
    const httpOptions = {
        method: 'POST',
        body: 'grant_type=client_credentials&scope=ALL',
        headers: {
            'Content-Type': 'application/x-www-form-urlencoded',
            Authorization: auth
        },
    };
    return XHR.fetch(authUri, httpOptions).then((x) => {
        const token =  JSON.parse(x.body);
        token.requestedAt = new Date();
        return token;
    });
}

Next up, we have another utility function that retrieves a cached API token from the PubNub KV store (if present), fetches a new token (if not present) and returns the token.

// See Mnubo API
function retrieveAccessToken(authUri, clientId, clientSecret) {
    const fetchAndStoreAccessToken = () => {
        return fetchAccessToken(authUri, clientId, clientSecret).then((newToken) => {
            const expiresInSeconds = newToken.expiresIn * 1000;
            return db.set('token', { value: newToken }, expiresInSeconds).then(() => newToken);
        });
    };
    return db.get('token').then((obj) => {
        const storedToken = obj.value;
        if (storedToken) {
            const now = new Date();
            const requestedAt = new Date(storedToken.requestedAt);
            const expiredTime = requestedAt.getTime() + storedToken.expires_in * 1000;
            const hasExpired = now.getTime() > expiredTime;
            if (hasExpired) {
                return fetchAndStoreAccessToken();
            } else {
                return storedToken;
            }
        } else {
            return fetchAndStoreAccessToken();
        }
    });
}

A third utility function takes care of sending event data to the mnubo Events API. It requires the token mentioned previously, and POSTs the event data to the correct API endpoint.

// See mnubo Events POST API
function sendEvents(eventsUri, token, events) {
    const httpOptions = {
        method: 'POST',
        body: JSON.stringify(events),
        headers: {
            'Content-Type': 'application/json',
            Authorization: `Bearer ${token.access_token}`
        },
    };
    return XHR.fetch(eventsUri, httpOptions);
}

Although we’ve reached the end of the code, here’s where the magic begins. We create an onRequest function that handles incoming messages. Make sure to change the API credentials and baseURI below to the ones from your trial (or production) account.

function onRequest(request) {
    const clientId = 'YOUR_CLIENT_ID';
    const clientSecret = 'YOUR_CLIENT_SECRET';
    const baseUri = 'https://rest.sandbox.mnubo.com';
    const authUri = `${baseUri}/oauth/token`;
    const eventsUri = `${baseUri}/api/v3/events`;

The main objective of the code below is to retrieve the access token, create a new event JSON object, and POST it to the mnubo API. If the result is successful, we decorate the message with the result data and publish it to the original channel by returning request.ok().

    return retrieveAccessToken(authUri, clientId, clientSecret)
        .then((token) => {
            const device = request.message.id;
            const timestamp = request.message.timestamp;
            const status = request.message.status;
            const events = [
                {
                    x_object: { x_device_id: device },
                    x_event_type: 'system_health',
                    x_timestamp: timestamp,
                    // TODO: add 'status' to your mnubo schema
                    // status: status
                }
            ];
            return sendEvents(eventsUri, token, events);
        }).then((r) => {
            request.message.result = r;
            return request.ok();
        }).catch((e) => {
            console.error(e);
            return request.ok();
        });
}
export default onRequest;

 

One thing to mention is that we’ve commented out the status data object above because the default schema doesn’t know about it. Feel free to add it to your own mnubo event schema to start capturing and analyzing it!

All in all, it doesn’t take a lot of code to add IoT events ingestion to our application. We like that!

OK, let’s move on to the UI!

Diving into the Code – the User Interface

You’ll want to grab these 100 lines of HTML & JavaScript and save them to a file, say, pubnub_mnubo_ui.html.

The first thing you should do after saving the code is to replace two values in the JavaScript:

  • YOUR_PUB_KEY: with the PubNub publish key mentioned above.
  • YOUR_SUB_KEY: with the PubNub subscribe key mentioned above.

If you don’t, the UI will not be able to communicate with anything and probably clutter your console log with entirely too many errors. For your convenience, this code is also available as a Gist on GitHub, and a Codepen as well. Enjoy!

Dependencies

First up, we have the JavaScript code & CSS dependencies of our application.

<!DOCTYPE html>
<html>
  <head>
    <title>Angular 2</title>
    <script src="https://unpkg.com/core-js@2.4.1/client/shim.min.js"></script>
    <script src="https://unpkg.com/zone.js@0.7.2/dist/zone.js"></script>
    <script src="https://unpkg.com/reflect-metadata@0.1.9/Reflect.js"></script>
    <script src="https://unpkg.com/rxjs@5.0.1/bundles/Rx.js"></script>
    <script src="https://unpkg.com/@angular/core/bundles/core.umd.js"></script>
    <script src="https://unpkg.com/@angular/common/bundles/common.umd.js"></script>
    <script src="https://unpkg.com/@angular/compiler/bundles/compiler.umd.js"></script>
    <script src="https://unpkg.com/@angular/platform-browser/bundles/platform-browser.umd.js"></script>
    <script src="https://unpkg.com/@angular/forms/bundles/forms.umd.js"></script>
    <script src="https://unpkg.com/@angular/platform-browser-dynamic/bundles/platform-browser-dynamic.umd.js"></script>
    <script src="https://unpkg.com/pubnub@4.3.3/dist/web/pubnub.js"></script>
    <script src="https://unpkg.com/pubnub-angular2@1.0.0-beta.8/dist/pubnub-angular2.js"></script>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" />
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css" />
  </head>

For folks who have done front-end implementation with Angular2 before, these should be the usual suspects:

  • CoreJS ES6 Shim, Zone.JS, Metadata Reflection, and RxJS : Dependencies of Angular2.
  • Angular2 : core, common, compiler, platform-browser, forms, and dynamic platform browser modules.
  • PubNub JavaScript client: to connect to our data stream integration channel.
  • PubNub Angular2 JavaScript client: provides PubNub services in Angular2 quite nicely indeed.

In addition, we bring in the CSS features:

  • Bootstrap: in this app, we use it just for vanilla UI presentation.

Overall, we were pretty pleased that we could build a nifty UI with so few dependencies. And with that… on to the UI!

The User Interface

Here’s what we intend the UI to look like:

The User Interface

The UI is pretty straightforward – everything is inside a main-component tag that is managed by a single component that we’ll set up in the Angular2 code.

<body>
  <main-component>
    Loading...
  </main-component>

Let’s skip forward and show that Angular2 component template. The h3 heading should be pretty self-explanatory. We provide a select box with values and a simple button to perform the publish() action to send an event to be processed by the BLOCK.

<div class="container">
    <pre>
    NOTE: make sure to update the PubNub keys below with your keys,
    and ensure that the BLOCK settings are configured properly!
    </pre>
    <h3>MyApp IoT Events Integration</h3>
    <hr/>
    <p>Click the button below to send a health metric</p>
    <select [(ngModel)]="system_health">
      <option value="healthy">healthy</option>
      <option value="degraded">degraded</option>
      <option value="disabled">disabled</option>
    </select>
    <button class="btn btn-primary btn-info" (click)="publish()">Send event!</button>
    <br/>
    <br/>
    <ul>
      <li *ngFor="let item of messages.slice()">
        <div>Device {{item.message.id}} @ {{item.message.timestamp}}, status = {{item.message.status}}, published {{item.message.result.statusText}}</div>
      </li>
    </ul>
</div>

The component UI consists of a simple list of events (in our case, the events data). We iterate over the messages in the component scope using a trusty ngFor. Each message includes the data from the mnubo API.

And that’s it – a functioning real-time UI in just a handful of code (thanks, Angular2)!

The Angular2 Code

Right on! Now we’re ready to dive into the Angular2 code. It’s not a ton of JavaScript, so this should hopefully be pretty straightforward.

The first lines we encounter set up our application (with a necessary dependency on the PubNub AngularJS service) and a single component (which we dub main-component).

<script>
var app = window.app = {};
app.main_component = ng.core.Component({
    selector: 'main-component',
    template: `...see previous...`

The component has a constructor that takes care of initializing the PubNub service and configuring the channel name, device id (you’d want your own to be set up using an application-specific method), and initial value. NOTE: make sure the channel matches the channel specified by your BLOCK configuration and the BLOCK itself!

}).Class({
    constructor: [PubNubAngular, function(pubnubService){
        var self = this;
        self.pubnubService = pubnubService;
        self.channelName = 'mnubo-post-event-channel';
        self.system_health = "healthy";
        self.device_id = PubNub.generateUUID();

Early on, we initialize the pubnubService with our credentials.

pubnubService.init({
    publishKey:   'YOUR_PUB_KEY',
    subscribeKey: 'YOUR_SUB_KEY',
    ssl:true
});

We subscribe to the relevant channel, create a dynamic attribute for the messages collection, and configure a blank event handler since the messages are presented unchanged from the incoming channel.

    pubnubService.subscribe({channels: [self.channelName], triggerEvents: true});
    self.messages = pubnubService.getMessage(this.channelName,function(msg){
      // no handler necessary, dynamic collection of msg objects
    });
}],

We also create a publish() event handler that performs the action of publishing the new message to the PubNub channel.

    publish: function(){
        var message = {
          id:this.device_id,
          timestamp:new Date().toISOString(),
          // TODO: be sure to to add 'status' to your mnubo event schema
          status:this.system_health
        };
        this.pubnubService.publish({channel: this.channelName,message: message});
    }
});

Now that we have a new component, we can create a main module for the Angular2 app that uses it. This is pretty standard boilerplate that configures dependencies on the Browser and Forms modules and the PubNubAngular service.

app.main_module = ng.core.NgModule({
    imports: [ng.platformBrowser.BrowserModule, ng.forms.FormsModule],
    declarations: [app.main_component],
    providers: [PubNubAngular],
    bootstrap: [app.main_component]
}).Class({
    constructor: function(){}
});

Finally, we bind the application bootstrap initialization to the browser DOM content loaded event.

document.addEventListener('DOMContentLoaded', function(){
    ng.platformBrowserDynamic.platformBrowserDynamic().bootstrapModule(app.main_module);
});

We mustn’t forget close out the HTML tags accordingly.

});
</script>
</body>
</html>

Not too shabby for about 100 lines of HTML & JavaScript!

Additional SmartObjects features include:

  • IoT Product Sessions: Track event data as a logical flow of interactions, like “connect, charge, disconnect”
  • IoT Asset Scoring: Score products based on usage and performance events to understand service and maintenance needs, different levels of user engagement, etc
  • IoT TS Predictions: Analyze forecasts of IoT timeseries data to enable proactive use cases such as; when to replace an industrial filter, when to replenish your fridge, etc
  • IoT Product Activity: Understand periods of activity for your smart products to enable appropriate business measures. For example, smart watch hasn’t been used in xx days, notify user of new feature
  • Custom Dashboards: Create custom dashboards with powerful visualizations and drill-downs for further analysis

All in all, we found it pretty easy to get started with the mnubo API and we look forward to using more of the integration features!

Conclusion

Thank you for joining us in the ‘IoT events’ episode of our ‘BLOCKS and web services’ series! Hopefully it’s been a useful experience learning about IoT-enabled technologies. In future articles, we’ll dive deeper into additional web service APIs and use cases for other nifty services.

Stay tuned, and please reach out anytime if you feel especially inspired or need any help!

0