QuantConnect Streams Financial Data to Realtime Dashboards Using PubNub

QuantConnect Streams Financial Data to Realtime Dashboards
QuantConnect is a browser-based algorithmic trading and investing backtesting platform. In finance, a backtester is a tool that is used to test out your ideas for an investment strategy. QuantConnect allows financial engineers, aka Quants, to code up their investment strategy and test it using historical data. In its coding environment, users can enter whatever code they want, select the data range, and the code is executed on that historical range. When the simulation is run, the results of the algorithm are shown as they play out in real-time.

Challenge

QuantConnect’s core product is the Backtester, which allows users to code their algorithms and backtest their strategies. One of the key features of Backtester is a real-time charting of the investment results as they happen and a fully interactive UI. When a user runs their algorithm, they can interact with and watch their investment strategy unfold in real-time as the data plots itself on a graph. But because users will often be running hundreds of different stock prices that require thousands of different data points, this amount of information is extremely heavy and demanding.

QuantConnect Charting

QuantConnect Charting (click to enlarge)

“It’s fairly demanding because there is a lot of data to show. There are a lot of charts, and there are a lot of orders to process. Some users generate thousands of orders with hundreds of stocks,” said Jared Broad, co-founder of QuantConnect.

With the core functionality lying in these real-time charts, QuantConnect needed a real-time network that could deliver and present an extremely heavy load of information and data in real-time to users. Running a backtest is long job, so QuantConnect wanted to be able to update the charts in real-time and show the algorithm’s progress as it runs.

The data packets required for this are very intensive and required a large amount of data to be sent through the network as quickly as possible. QuantConnect needed a way to deliver these heavy data packets through a real-time network. Originally unable to pass the chart data because of issues with packet size, the development team tried instead to pass the location of a file which contained the chart data. On the client side, every time they received an updated packet location the browser would redownload the updated chart data and display it.

“When you start to analyze gigabytes of data, its quite an intensive job for the messaging system and was pretty messy. PubNub was being used as a data flag, signalling a download of a JSON file and that download was causing backtester and sometimes user’s web browsers to freeze and crash,” said Broad.

Solution

QuantConnect originally looked at WebSockets, an open source solution, and another real-time service provider before picking the PubNub Real-Time Network. In under a week, Backtester had integrated the PubNub into the application and rebuilt their API around it.

QuantConnect took a couple steps to resolve with the large packet issue. First, they had to build their JavaScript API on top of PubNub messaging system. Then, they built an intelligent buffer system that cached the data until it reached PubNub message size limits, and then processed the packets when it was at the message limit size. This stopped crashing the user’s browsers and significantly increased the speed of the app.

“With the revamp, I took this charting data and broke it into much smaller packets, added up the length of those packets until the data was roughly the right size for PubNub. We then synced the packet when it was the right size. Using the PubNub maximum size of 7200 characters, I built up a buffer until we reached 7200, then we synched, cleared the buffer, then started again,” said Broad.

QuantConnect Slide Trades

QuantConnect Trades (click to enlarge)

Instead of using PubNub purely for signalling in order to pull from the servers, QuantConnect increased the message size and used throttling in order to bunch aggregate data to get to the optimal large message size (7.2k). This reduced the rate for publishing through PubNub. It came down to optimizing how they packed data, so that the messaging publishing rate is reduced.

Results

Integrating PubNub with QuantConnect significantly increased both the speed and stability of the Backtester. Because they were able to reduce the weight of each data packet and efficiently apply a wrapper, the development team found that the app tripled in speed.

“When we started, first the signal would arrive, then we would go request the data, then the data would come back. Now the data arrived directly from PubNub. It is exactly three times as fast.”

Looking at stability, Quant Connect was able to use lower server resources and make the average browser run faster and smoother.

“Processing smaller packets released control to the javascript UI thread more frequently, resulting in less processing time, and lower RAM usage. As browsers are a highly memory and resource constrained environment, high RAM or processor usage causes browser crashes in some older PC’s. This made our system 10x as stable,” said Broad.

Try PubNub Today