I’ve had a fair number of conversations with our customers to develop PubNub case studies, and I’ve deduced a common theme embedded in each of their stories. Most of them have first experimented with building out a real-time data stream network infrastructure in-house, and have run into a series of common roadblocks in doing so. This blog post will categorize the issues you may run into when building a real-time data stream infrastructure on your own.
Before we dive into this blog post, it’s important to first define what it means to build and what it means to buy.
Build: Building a data stream network on your own means you’re taking an open source protocol, tool, and framework that already exists in an open source community (like Socket.IO, SignalR, etc), then installing and orchestrating the operations of that open source service (into physical or virtualized hardware) while maintaining platform enhancements.
We love, respect and embrace the spirit of DIY innovation. After all, that’s how PubNub was born. Installing and operating a socket server for the first time is fun and fulfilling. However as scale increases, the DIY solution typically fails due to orchestration challenges.
The purpose of this blog post is not to discourage building a real-time data stream network on your own. Rather, it’s to point out some of the issues you may face if you elect to build it yourself, and share how many developers have gotten past those issues by using a real-time data stream network service provider.
Often one of the first things that might go through your mind when you’re looking into whether to build vs buy is the idea of upfront cost; how much will it cost to build it yourself from scratch versus buying it from a real-time data stream service provider? As you well know, though, it’s not just the initial cost, but the total cost down the road. Here are a few things to consider when estimating those initial and down-the-road costs.
There’s not much code you have to write to get a real-time system set up other than some ops code (real-time open source protocols and frameworks are pretty streamlined these days). When you start with an open source protocol or framework, you can carry out basic real-time functionality on your socket-type broadcast server. These functions include publish/subscribe, unicast and broadcast messaging.
But building out a fully functional data stream network on your own isn’t as easy or cheap as you may think. There are a number of considerations and costs that need to be taken into account when building it yourself. It’s important to weigh the total cost of ownership for your solution.
If a primary concern of buying a service is upfront cost, it shouldn’t be. Many real-time service providers provide a free sandbox developer tier where you can build and test your real-time app free of charge, for as long as you need to.
When you decide to move your app from the lab and deploy it into production, scale and orchestration need to be taken into consideration. When you use a data stream network service provider, they handle this in all its complexity so you don’t have to. It’s their job to set up and maintain servers, install and update software and versions, upgrade features and add new service capabilities, and perform any and all necessary maintenance.
You may also run into the issue of picking the correct framework or protocol for your exact need. A data stream service provider will oftentimes build a number of SDKs available for whatever platform or device you want your app to run on. These SDKs are MIT open source, and can be modified as needed. They are also monitored and updated continuously.
Reliability and scalability are critical. Reliability of your data stream network is essential for ensuring a consistent, fast real-time user experience. This means that when deploying and scaling apps to thousands of users simultaneously, you need a global presence; data centers must proliferate across different regions of Earth. If you’re targeting a certain region exclusively, one data center is able to handle that kind of user base. Otherwise, it’s important to make sure the network can scale easily when launched.
Redundancy is another key component. In the event that your data centers go down, your data stream network goes down. Data stream network service providers offer globally redundant data stream networks, so if one data center degrades, all real-time data has already been replicated to all other data centers in different regions, ensuring delivery of all messages.
There are common underlying technologies for a dating app, and in this post, we’ll talk about the major technologies and designs...