QuantConnect is a browser-based algorithmic trading and investing backtesting platform. In finance, a backtester is a tool that is used to test out your ideas for an investment strategy. QuantConnect allows financial engineers, aka Quants, to code up their investment strategy and test it using historical data. In its coding environment, users can enter whatever code they want, select the data range, and the code is executed on that historical range. When the simulation is run, the results of the algorithm are shown as they play out in real-time.
QuantConnect’s core product is the Backtester, which allows users to code their algorithms and backtest their strategies. One of the key features of Backtester is a real-time charting of the investment results as they happen and a fully interactive UI. When a user runs their algorithm, they can interact with and watch their investment strategy unfold in real-time as the data plots itself on a graph. But because users will often be running hundreds of different stock prices that require thousands of different data points, this amount of information is extremely heavy and demanding.
“It’s fairly demanding because there is a lot of data to show. There are a lot of charts, and there are a lot of orders to process. Some users generate thousands of orders with hundreds of stocks,” said Jared Broad, co-founder of QuantConnect.
With the core functionality lying in these real-time charts, QuantConnect needed a real-time network that could deliver and present an extremely heavy load of information and data in real-time to users. Running a backtest is long job, so QuantConnect wanted to be able to update the charts in real-time and show the algorithm’s progress as it runs.
The data packets required for this are very intensive and required a large amount of data to be sent through the network as quickly as possible. QuantConnect needed a way to deliver these heavy data packets through a real-time network. Originally unable to pass the chart data because of issues with packet size, the development team tried instead to pass the location of a file which contained the chart data. On the client side, every time they received an updated packet location the browser would redownload the updated chart data and display it.
“When you start to analyze gigabytes of data, its quite an intensive job for the messaging system and was pretty messy. PubNub was being used as a data flag, signalling a download of a JSON file and that download was causing backtester and sometimes user’s web browsers to freeze and crash,” said Broad.
QuantConnect originally looked at WebSockets, an open source solution, and another real-time service provider before picking the PubNub Real-Time Network. In under a week, Backtester had integrated the PubNub into the application and rebuilt their API around it.
“With the revamp, I took this charting data and broke it into much smaller packets, added up the length of those packets until the data was roughly the right size for PubNub. We then synced the packet when it was the right size. Using the PubNub maximum size of 7200 characters, I built up a buffer until we reached 7200, then we synched, cleared the buffer, then started again,” said Broad.
Instead of using PubNub purely for signalling in order to pull from the servers, QuantConnect increased the message size and used throttling in order to bunch aggregate data to get to the optimal large message size (7.2k). This reduced the rate for publishing through PubNub. It came down to optimizing how they packed data, so that the messaging publishing rate is reduced.
Integrating PubNub with QuantConnect significantly increased both the speed and stability of the Backtester. Because they were able to reduce the weight of each data packet and efficiently apply a wrapper, the development team found that the app tripled in speed.
“When we started, first the signal would arrive, then we would go request the data, then the data would come back. Now the data arrived directly from PubNub. It is exactly three times as fast.”
Looking at stability, Quant Connect was able to use lower server resources and make the average browser run faster and smoother.