Free up to 1MM monthly messages. No credit card required.
Imagine a world where you walk into your home, look at a smart lightbulb in the room, make a gesture with your hands, and the light turns off or on. Or you swipe with a hand to change the channel on your TV. Or you even glance at a thermostat and speak your preferred temperature. This isn’t the future. This is all possible now with the Magic Leap One and PubNub.
This is part two of our Magic Leap series. Check out the other posts Getting Started with Magic Leap and PubNub and Create a Multiplayer Augmented Reality Game with Magic Leap and Unity.
In this tutorial, we will walk through how to interface with internet-connected devices using the Unity Video Game Engine for Magic Leap and the PubNub Unity SDK. In this case, we’ll turn on and off, and change the color of an internet-connected lightbulb.
Be aware that the Magic Leap SDKs are changing as new features are added and some parts of this tutorial may change. The following steps were tested with Lumin SDK Version 0.17.0 and Unity Version: 2018.1.9f1-MLTP8.1.
Looking for the completed project? Download it from the GitHub repo. In addition to the gesture recognition from this post, the finished project has a UI that indicates the gesture you’re using and implements a custom vignette.
Before continuing with this tutorial, you should read the Getting Started with Magic Leap and Unity tutorial first to familiarize yourself with Unity Video Game Engine development for Magic Leap and setup your development environment.
There are a few requirements before you can get started:
#if((!USE_JSONFX_UNITY_IOS) && (!USE_MiniJSON)) //#define USE_JSONFX_UNITY_IOS #define USE_MiniJSON #endif
You first need to import the Magic Leap Unity Package in order to use the Hand Tracking API.
Check out the Hand Tracking, Hand Tracking – Key Points in Unity®, and Hand Poses in Unity® guides by Magic Leap to learn more about using hand poses and hand key points in your apps.
using System.Collections; using System.Collections.Generic; using UnityEngine; using UnityEngine.UI; using UnityEngine.XR.MagicLeap; using PubNubAPI; public class Gestures : MonoBehaviour { public float sendTimeController; public static PubNub pubnub; private MLHandKeyPose[] gestures; // Holds the different hand poses we will look for. void Awake () { PNConfiguration pnConfiguration = new PNConfiguration(); pnConfiguration.PublishKey = "YOUR_PUB_KEY_HERE"; pnConfiguration.SubscribeKey = "YOUR_SUB_KEY_HERE"; pnConfiguration.Secure = true; pubnub = new PubNub(pnConfiguration); MLHands.Start(); // Start the hand tracking. gestures = new MLHandKeyPose[3]; //Assign the gestures we will look for. gestures[0] = MLHandKeyPose.Fist; gestures[1] = MLHandKeyPose.Thumb; gestures[2] = MLHandKeyPose.Finger; MLHands.KeyPoseManager.EnableKeyPoses(gestures, true, false); // Enable the hand poses. } void OnDestroy() { MLHands.Stop(); } void Update() { if (sendTimeController <= Time.deltaTime) { // Restrict how often messages can be sent. if (GetGesture(MLHands.Left, MLHandKeyPose.Thumb) || GetGesture(MLHands.Right, MLHandKeyPose.Thumb)) { pubnub.Publish() .Channel("control") .Message("on") .Async((result, status) => { if (!status.Error) { Debug.Log(string.Format("Publish Timetoken: {0}", result.Timetoken)); } else { Debug.Log(status.Error); Debug.Log(status.ErrorData.Info); } }); sendTimeController = 0.1f; // Stop multiple messages from being sent. } else if (GetGesture(MLHands.Left, MLHandKeyPose.Fist) || GetGesture(MLHands.Right, MLHandKeyPose.Fist)) { pubnub.Publish() .Channel("control") .Message("off") .Async((result, status) => { if (!status.Error) { Debug.Log(string.Format("Publish Timetoken: {0}", result.Timetoken)); } else { Debug.Log(status.Error); Debug.Log(status.ErrorData.Info); } }); sendTimeController = 0.1f; // Stop multiple messages from being sent. } else if (GetGesture(MLHands.Left, MLHandKeyPose.Finger)) { pubnub.Publish() .Channel("control") .Message("changel") .Async((result, status) => { if (!status.Error) { Debug.Log(string.Format("Publish Timetoken: {0}", result.Timetoken)); } else { Debug.Log(status.Error); Debug.Log(status.ErrorData.Info); } }); sendTimeController = 0.9f; // Stop multiple messages from being sent. } else if (GetGesture(MLHands.Right, MLHandKeyPose.Finger)) { pubnub.Publish() .Channel("control") .Message("changer") .Async((result, status) => { if (!status.Error) { Debug.Log(string.Format("Publish Timetoken: {0}", result.Timetoken)); } else { Debug.Log(status.Error); Debug.Log(status.ErrorData.Info); } }); sendTimeController = 0.9f; // Stop multiple messages from being sent. } } else { sendTimeController -= Time.deltaTime; // Update the timer. } } bool GetGesture(MLHand hand, MLHandKeyPose type) { if (hand != null) { if (hand.KeyPose == type) { if (hand.KeyPoseConfidence > 0.98f) { return true; } } } return false; } }
PubNub offers an Arduino SDK that we will use to control an RGB LED via the Magic Leap One.
http://arduino.esp8266.com/stable/package_esp8266com_index.json
#include <ESP8266WiFi.h> #define PubNub_BASE_CLIENT WiFiClient #include <PubNub.h> const static char ssid[] = "YOUR_SSID"; const static char pass[] = "YOUR_PASSWORD"; int rled = 14; // The PWM pins the LED is attached to. int gled = 12; int bled = 15; int mode = 0; bool on = false; void setup() { pinMode(rled, OUTPUT); pinMode(gled, OUTPUT); pinMode(bled, OUTPUT); Serial.begin(9600); WiFi.begin(ssid, pass); if(WiFi.waitForConnectResult() == WL_CONNECTED){ PubNub.begin("YOUR_PUB_KEY_HERE", "YOUR_SUB_KEY_HERE"); } else { Serial.println("Couldn't get a wifi connection"); while(1) delay(100); } } void loop() { PubNub_BASE_CLIENT *client; Serial.println("waiting for a message (subscribe)"); PubSubClient *pclient = PubNub.subscribe("control"); // Subscribe to the control channel. if (!pclient) { Serial.println("subscription error"); delay(1000); return; } String message; while (pclient->wait_for_data()) { char c = pclient->read(); message = message+String(c); // Append to string. } pclient->stop(); if(message.indexOf("on") > 0) { on = true; Serial.print("on"); } else if (message.indexOf("off") > 0) { on = false; } else if (message.indexOf("changel") > 0) { if (mode > 0) { mode=mode-1; } else if (mode == 0) { mode=5; } } else if (message.indexOf("changer") > 0) { if (mode < 5) { mode=mode+1; } else if (mode == 5) { mode=0; } } if (on == true) { // Turn on led. if (mode == 0) { // White analogWrite(rled, 255); analogWrite(gled, 255); analogWrite(bled, 255); } else if (mode == 1) { // Less Bright White analogWrite(rled, 255); analogWrite(gled, 165); analogWrite(bled, 0); } else if (mode == 2) { // Red analogWrite(rled, 255); analogWrite(gled, 0); analogWrite(bled, 0); } else if (mode == 3) { // Green analogWrite(rled, 0); analogWrite(gled, 255); analogWrite(bled, 0); } else if (mode == 4) { // Blue analogWrite(rled, 0); analogWrite(gled, 0); analogWrite(bled, 255); } else if (mode == 5) { // Purple analogWrite(rled, 255); analogWrite(gled, 0); analogWrite(bled, 255); } } else { // Turn off led analogWrite(rled, 0); analogWrite(gled, 0); analogWrite(bled, 0); } Serial.print(message); Serial.println(); delay(5); }
There are three ways to test your app:
See the Launch Magic Leap Remote and Play Mode with Magic Leap Remote guides from Magic Leap for more information about using the Magic Leap Remote.
You can change your code and see the results immediately while in play mode.
If you do not have a device, play mode simulates the Lumin SDK API layers and composites the graphics with input from a virtual room.
If you do have a device, play mode simulates the Lumin SDK API layers and then streams the rendered frames to the headset display.
You won’t see anything if you look around in the simulator on a device. However, messages are being sent to PubNub while the app is running whenever you make an enabled gesture. The finished project has a UI that indicates the gesture you’re using. Download the complete project from the GitHub repo.
If you’re using the simulator: Read the Magic Leap guide on simulating input on magic leap remote to simulate hand gestures.
To use the Magic Leap Remote you will need to enable Magic Leap zero iteration mode. In Unity click “MagicLeap” and then “Enable Zero Iteration”. Restart the editor.
See Deploying a Unity App to the Device from Magic Leap for more information about deploying an app a device.
We did not build a UX for this app as this post focuses on detecting gestures and sending messages to PubNub. You won’t see anything if you look around in the simulator on a device. However, messages are being sent to PubNub while the app is running whenever you make an enabled gesture. The finished project has a UI that indicates the gesture you’re using. Download the complete project from the GitHub repo.
Finger Gesture Changing LED Color Mode.
Use this project as a seed to build your own Magic Leap IoT application. A few ideas:
Power strip control with Magic Leap IoT App.
This is part two of our Magic Leap series. Check out the other posts Getting Started with Magic Leap and PubNub and Create a Multiplayer Augmented Reality Game with Magic Leap and Unity.
A Notice of Privacy Practices (NPP) is one of the requirements of HIPAA and helps patients understand their personal data rights.
Michael Carroll
HIPAA violations can be financially expensive and devastating to a brand. Examine some examples of HIPAA violations, and learn...
Michael Carroll
HIPAA covered entities must follow the five technical safeguards to achieve HIPAA compliance and prevent data corruption.
Michael Carroll