Create a Real-time RSS Feed Using Python and JavaScript

3 min readPubNub Staff on Jun 11, 2014
Try PubNub Today

Free up to 1MM monthly messages. No credit card required.

Subscribe to our newsletter

By submitting this form, you are agreeing to our Terms and Conditions and Privacy Policy.

How to build a real-time RSS feed using Python and JavaScript, pushing content to a browser in real time as it's published

Madison Smith is a intern at PubNub, who’ll be cooking up awesome demos and projects all summer, including this real-time RSS feed with Python and JavaScript.

Real-time rss feed

Social news websites have changed the way we find, read, and share content. However, these websites are static and fresh content is not presented as it happens. To this, I aimed to change this, and I picked Hacker News as my news source. By leveraging the power of PubNub’s global data stream network, and scraping a little RSS, nobody will ever miss a new Hacker News article again.

I built a news and social content feed, that auto updates in real time as new content is posted to Hacker News. However, this can be applied to pretty much any social news website (all you need is a RSS feed). This otherwise static content is now pushed in real time to the browser.

If you want to see it working live, check out the Hacker News Feed Demo. It uses the JavaScript PubNub SDK and will display the updates to the Hacker News feed. To see it in action locally, clone the source from Github and run the Python scraper from the command line.

RSS Scraping

You’ll first need to sign up for a PubNub account. Once you sign up, you can get your unique PubNub keys in the PubNub Developer Portal.

The first task is to grab the RSS feed from Hacker News. There is a plethora of ways to do this and you can quickly write your own RSS scraper if you want, but I decided to use Python and feedparser. With a quick “pip install feedparser” we have our RSS.

With no customization, you’ll get every single post that’s posted. However, I decided the most interesting information was the rank of the post, title of the post, the link to the article, and the comments link.

Python Command Line

The Python Argparse module is used, which very powerfully gives you robust command line options.

You can python --help to see descriptions of all the options from the command line. The Python module gives you options for specifying how often you want to poll Hacker News for changes and if you want to get a new page after every change to the site or just the new posts that appear on the site. For instance, if you wanted to poll every five seconds and get the entire page you could run to be up and going:

Argparse also gives defaults, so run the following to use the defaults:

Go Global

Now that we have the information that is important to us, and know how to run the scraper locally, it’s time to send it global. PubNub provides our incredibly simple API to publish the message. Quickly “pip install Pubnub” and publish our information from Hacker News.

Now it’s up to you. PubNub offers over 50 different SDK’s for your use. Take your pick. When trying to consume the information simply subscribe to the channel (in our case “hacker-news”) and you’re off. There are publicly available publish and subscribe keys to use for demos.

And the final result should look something like this:

More from PubNub

Can Empathy Exist in the Metaverse
News May 16, 20221 min read

Can Empathy Exist in the Metaverse

A roundtable discussion led by PubNub’s COO, Casey Clegg, exploring the topics of what it means to be human in a virtual world.

PubNub Staff

PubNub Staff

How to Advance Telehealth and Virtual Care Technologies
News May 2, 20221 min read

How to Advance Telehealth and Virtual Care Technologies

Dr. Joe Kvedar, Chair of the Board for the American Telemedicine Association, joins our COO, Casey Clegg, to discuss why...

PubNub Staff

PubNub Staff

Another Step Towards Data Security: ISO-27001 Implementation
BuildMay 2, 20221 min read

Another Step Towards Data Security: ISO-27001 Implementation

Today, we are glad to announce that we are currently in the process of implementing ISO-27001 security standards.

PubNub Staff

PubNub Staff