DOCS
Ask AI

Stream Kafka records to PubNub clients

In this tutorial, you run the PubNub Kafka Sink Connector locally, then watch records from the Kafka topic pubnub arrive on the PubNub channel pubnub. The connector project supplies Kafka, Kafka Connect, and a producer that writes sample records to the topic.

Before you begin​

You need:

  • Docker Desktop or another Docker installation that includes Docker Compose.
  • A PubNub account and a keyset. A keyset is the set of publish, subscribe, and secret keys that identifies your application to the PubNub network. You need its publish key and subscribe key. If you do not have a keyset, set up your account first.
  • A local port 8083 available for Kafka Connect.

The connector runs in your Docker environment, not in a client application. Keep your keyset credentials in the connector configuration and out of source control. The standard message payload size limit is 32 KiB. This includes the channel name and any metadata. If you replace the sample producer, keep each message under that limit.

Clone the connector project​

Clone the maintained connector project, then move into it:

git clone https://github.com/pubnub/pubnub-kafka-sink-connector.git
cd pubnub-kafka-sink-connector

The project uses the topic name pubnub throughout this tutorial. The connector's default mapping publishes records from that topic to a PubNub channel with the same name.

Add your PubNub keys​

Open examples/pubnub-sink-connector.json. It is the Kafka Connect configuration that creates the connector.

1

Replace these three values:

  • pubnub.user_id with kafka-connect-tutorial.
  • pubnub.publish_key with the publish key from your keyset.
  • pubnub.subscribe_key with the subscribe key from your keyset.

Leave topics set to pubnub. The included producer writes records to that topic, and the connector publishes them to the PubNub channel with the same name.

Leave pubnub.secret_key set to demo. This tutorial does not use Access Manager, so the connector does not need your keyset's secret key.

Start Kafka and Kafka Connect​

Start the Docker Compose project in the background:

docker compose up -d

Docker builds the connector, starts Kafka and Kafka Connect, and starts the sample producer. Wait until Kafka Connect responds:

curl --fail http://localhost:8083/connectors

The first response is an empty JSON array:

[]

If the request cannot connect yet, wait a few seconds and run it again. Kafka Connect does not accept the connector configuration until the service has started.

Deploy the connector​

Send the configuration file to Kafka Connect:

curl --fail-with-body \
  --request POST \
  --header "Content-Type: application/json" \
  --data @examples/pubnub-sink-connector.json \
  http://localhost:8083/connectors

Kafka Connect returns JSON that includes the connector name pubnub-sink-connector. The sample producer is already writing records such as this to the pubnub topic:

{ "timestamp": 1705689453 }

The connector publishes each record value as a PubNub message on the channel pubnub.

Verify the message arrives​

Open the PubNub Debug Console. Enter the publish key and subscribe key from the keyset you configured. Set the channel to pubnub and choose any User ID, such as kafka-tutorial-observer.

Messages begin appearing on the channel:

{ "timestamp": 1705689453 }

The timestamp changes as the sample producer writes new Kafka records. Seeing those messages confirms the full path:

Kafka topic pubnub → Kafka Connect → PubNub channel pubnub → Debug Console

What happened​

  1. Docker Compose started a Kafka broker, Kafka Connect, the PubNub connector build, and a sample Kafka producer.
  2. Kafka Connect loaded the PubNub Kafka Sink Connector with your keyset credentials.
  3. The connector read records from the pubnub Kafka topic and published their values to the PubNub channel pubnub.
  4. Debug Console subscribed to that channel and displayed the live messages.

Stop the tutorial environment​

Remove the connector, then stop the containers:

curl --fail --request DELETE http://localhost:8083/connectors/pubnub-sink-connector
docker compose down

Next steps​

Was this page useful?

Last updated on