If you have an existing Kafka cluster and you're looking to deliver data records from your Kafka topics to devices such as web browsers and mobile phones in real-time, PubNub Kafka Sink Connector is the solution. This guide will walk you through the process of setting up the PubNub Kafka Sink Connector.
Setting up a Local Test Kafka Cluster (optional)
The PubNub Kafka Sink Connector leverages Kafka Connect, a framework developed specifically for delivering data from Kafka topics to secondary systems such as Hadoop, Elastic, and PubNub. For demonstration purposes, we've included an entire Kafka cluster in a single Docker composed file. It includes everything you need to set up your Docker, the composed YAML files, Kafka broker, and the cluster itself. Additionally, we've included a producer for testing purposes that automatically produces data into a topic on your locally running Kafka cluster.
Clone the repository https://github.com/pubnub/pubnub-kafka-sink-connector and run the command docker compose up to boot these images on your local system. This might take a while, as there's a lot to Kafka itself.
The Docker Compose will also build the PubNub Kafka Sink Connector package. We've included an image that will build the Maven package for us for simplicity purposes. This makes it quick and easy to import into your Kafka Connect using the Kafka Connect framework. Once the builder image has finished building the connector package, it's ready for deployment. You'll know it's ready when the consumer notification messages appear at regular intervals showing: consumer | {"timestamp":1705690559} in a series of records. This represents that our test data container is operating successfully and we are ready to install the Sink Connector.
Deploying the PubNub Sink Connector
The Sink Connector is the bridge running on the Kafka Connect framework between the Kafka topics and the records in those topics over to your devices. All you need is the connector package information, which is in the PubNub Sink Connector JSON file. Post this file to Kafka Connect where Kafka Connect is running, and it will begin copying data from the configured topics directly over to your end target devices.
You need to configure the JSON file examples/pubnub-sink-connector.json with your PubNub API key information. You can find your API keys at https://admin.pubnub.com/ This is what's going to be used to deliver the end results to your target devices. You can set your topics and your patterns, and then run the command to deploy the Sink Connector.
Run the following command to deploy the sink connector:
Your connector is now running and all data sent to the configured Kafka topic will be instantly copied to any device subscribed with the PubNub SDK.
Validating the Sink Connector
You can validate that the connector is running by checking the messages being delivered to a web browser device. As soon as messages are put onto the Kafka topic, they are immediately copied over to the PubNub infrastructure, which is responsible for reliably delivering those messages to your end target devices. The following screenshot shows data streaming into a web browser application instantly the moment an event record is received in the Kafka topic.
Building PubNub Kafka Sink Connector
When you're ready for production, you can build PubNub Kafka Sink Connect using the Maven build command. Ensure you have all the dependencies installed such as Java 11 or greater and a new version of Maven. Run mvn clean package to run all the tests associated with the package and compile a couple of packages including a jar file and a zip file. This is the package that will be used in the Kafka Connect Framework.
Video Walkthrough for Installing PubNub Kafka Sink Connector
In the video a guided installation of the PubNub Kafka Sink Connector is demonstrated.
Setting up the PubNub Kafka Sink Connector is straightforward, and it's important to follow the steps carefully. If you encounter challenges along the way, reach out to us at support.pubnub.com, and we'll be happy to assist.
For more details, head to the official PubNub documentation.