Develop Multiplayer Experiences with Apple ARKit and World Mapping

By definition, Augmented Reality (AR) is technology that superimposes a computer-generated image on the user’s view of the physical world. In 2016, Pokemon Go kickstarted the augmented reality revolution and generated massive player involvement. This cultural phenomenon opened peoples eyes to the potential AR has to offer.

With the release of Apple’s ARKit 2, connecting devices together to share objects in the same virtual space became easy with the Share World Map button.  However, there is still a lack of support for realtime multiplayer augmented reality experiences with devices connected together from across the globe.  PubNub’s APIs and infrastructure can help solve these problems for developers by providing a JSON Pub/Sub architecture that scales to support millions of simultaneously connected devices globally.

In this post, I interviewed Arvin Tehrani and Steffanie Fowler from Contrast Media Labs, a development company using PubNub to facilitate realtime experiences in augmented reality.

To learn how they created this magic AR giraffe, read the interview and check out the code tutorial below.

When did both of you start Contrast Media Labs? What type of applications have you specialized in?

We started Contrast Media Labs in 2010 as a traditional web development company, throughout the years we’ve added iOS, Android and VR/ AR development. We currently focus on the enterprise space – helping companies utilize custom applications to reach their goals specifically using immersion and 3D.

We develop applications for companies as well as provide consultation. We’ve worked with large enterprises such as PG&E as well as startups comprised of a couple founders. In our experience, we’ve found that once companies discover the power of AR/VR, they find new ways to adopt the technology in other departments or areas of their businesses, which helps propel the adoption of these new technologies into mainstream use.

What application are you building with ARKit and PubNub? What audience are you targeting?

The current application we are working on with ARKit and PubNub is a concept utilizing AR in education. We are targeting schools with online or remote classrooms. We believe AR is going to be important in the educational space especially as AR hardware continues to advance.

When it comes to AR Kit, what were some challenges you ran into?

Initially, when ARKit was announced, the challenge was getting a multi-user experience that worked reliably. With the release of iOS 12, Apple has essentially solved that issue and that has opened up new possibilities with AR.

In your AR Kit application, you added realtime chat powered by PubNub. How was your overall experience using the PubNub API and why is realtime chat important to your users?

Our experience using PubNub was fantastic. We integrated PubNub into an iOS application using Swift as well as a React application for the web. The SDKs were well-documented and up-to-date.

Realtime chat allows the students to interact with the experience from a remote location which is very important for the experience to work. A fast and reliable pub/sub system is what we were searching for and PubNub delivered. The added bonus of not needing to manage and scale infrastructure allows us freedom in these early stages to focus on the more important features such as UX and network performance.

Do you have any recommendations on how developers can learn about this emerging technology? Can you provide any resources that helped you become successful?

Apple does a good job of documenting ARKit and the WWDC videos are a great place to start if you’re looking to learn. We find it important to follow other developers and artists working with ARKit and ARCore. The AR community is producing some great experiences using ARKit and ARCore, seeing what others are doing helps motivate us and pushes the industry forward.

Magic Giraffe ARKit + PubNub Demo

The giraffe demo was created using Apple’s ARKit and PubNub’s realtime APIs and infrastructure.  In the video below, the user can talk to the giraffe in order to make the giraffe respond in realtime on both displays.

The service starts with an iOS client that is hosting the experience. Other iOS clients can connect to the host to receive updates including the ARWorldMap to help iOS understand the environment as well as when to trigger animations.

The hosting iOS client is the only iOS device that connects to PubNub. It is responsible for sending updates to the connected web clients. The web client, which is a React application with the PubNub SDK, interacts with the experience from a remote location, while the iOS clients interact with the experience in the same room. This allows a person in a remote location to ask a question which can be addressed by the person hosting the experience using AR and the resulting video of the AR experience is streamed to the web clients.

iOS (Swift): Receiving Messages

func didReceiveMessage(message: String, from: String) {
      if !message.isEmpty && isHosting {
          questionProcessor.addQuestion(questionText: message, from: from)
      }
}

We’ve built a wrapper around the PubNub SDK to handle sending and receiving messages which allows us to add custom validation logic. This delegate function didReceiveMessage is called when a message from PubNub is received and passes the custom validation logic.

When we receive a message from PubNub in the iOS client, the first thing we do is check if the publisher is this same client and if so, we ignore the message to avoid a duplicate question since this application will send the current question to PubNub for the web clients and that message is broadcast to everyone subscribed. The iOS application contains a class that manages the questions queue.

Once we receive a valid question and we are hosting a session, we add the question to the queue to be processed. When the question is processed, it displays in the AR experience and the hosting iOS application sends the question to PubNub for the web clients. Click the button below to signup for a free PubNub account to generate your own Pub and Sub keys.

Web (JS/React): Sending Messages

// Intialize PubNubReact with publish key and subscribe key and then pass in the React component during the React Component's initialization
this.pubnub = new PubNubReact({
  publishKey: PUBLISH_KEY,
  subscribeKey: SUBSCRIBE_KEY
});
this.pubnub.init(this);

// During componentWillMount subscribe to the PubNub channel

componentWillMount() {
  this.pubnub.subscribe({
    channels: [CHANNEL_NAME],
    withPresence: false
  });
}

// Unsubscribe on unmount

componentWillUnmount() {
  if (this.pubnub) {
    this.pubnub.unsubscribe({
      channels: [CHANNEL_NAME]
    });
  }
}

// Form submit event
onQuestionSubmit(event) {
  event.preventDefault();
  // We're storing the question in the component state
  if (this.state.question === "") {
     return; 
  }
  // Ensure question is valid using our custom validation logic then publish the message
  if (this.isQuestionValid()) {
    // Send question and reset state
    this.publishMessage(this.state.question);
    this.setState({ question: ""});
  } 
}

// Sends a message to PubNub channel
publishMessage(message) {
  this.pubnub.publish({
      message: { text: message, from: this.state.username},
      channel: CHANNEL_NAME
  });
}

The React component that is responsible for sending messages uses the PubNubReact package. The component contains a simple form with an input for the question and a submit button. When the form is submitted, the onQuestionSubmit function is called and a message is published to the PubNub channel for the iOS client that is hosting the experience.

The React application only cares about received messages from the hosting iOS device (it ignores messages from other web clients). When it receives a question from the hosting iOS device, (which is identified by a pre-determined identifier) we display the message in the browser on top of the video feed.

Learn More

To learn more about Contrast Media Labs, go check out their website here and you can reach out to the Co-Founder Arvin Tehrani at arvin@contrastmedialabs.com.

If you have any questions about PubNub, and how to integrate our API’s into your augmented reality application, reach out to schuetz@pubnub.com for guidance.

Try PubNub Today