---
source_url: https://www.pubnub.com/docs/integrations/connect-pubnub-to-kafka
title: Stream Kafka records to PubNub clients
updated_at: 2026-09-30T07:20:08.000Z
---

# Stream Kafka records to PubNub clients

## Documentation index

To discover more PubNub resources:

1. Fetch [PubNub's llms.txt](https://www.pubnub.com/llms-full.txt) for a list of available pages in Markdown format.
2. Identify relevant URLs from that index.
3. Fetch the target pages.

Do not assume a path exists, always check the index first.

In this tutorial, you run the PubNub Kafka Sink Connector locally, then watch records from the Kafka topic `pubnub` arrive on the PubNub channel `pubnub`. The connector project supplies Kafka, Kafka Connect, and a producer that writes sample records to the topic.

## Before you begin

You need:

* Docker Desktop or another Docker installation that includes Docker Compose.
* A PubNub account and a keyset. A keyset is the set of publish, subscribe, and secret keys that identifies your application to the PubNub network. You need its publish key and subscribe key. If you do not have a keyset, [set up your account](https://www.pubnub.com/docs/architecture/authentication/set-up-your-account.md) first.
* A local port `8083` available for Kafka Connect.

The connector runs in your Docker environment, not in a client application. Keep your keyset credentials in the connector configuration and out of source control. The standard message payload size limit is 32 KiB. This includes the channel name and any metadata. If you replace the sample producer, keep each message under that limit.

## Clone the connector project

Clone the maintained connector project, then move into it:

```bash
git clone https://github.com/pubnub/pubnub-kafka-sink-connector.git
cd pubnub-kafka-sink-connector
```

The project uses the topic name `pubnub` throughout this tutorial. The connector's default mapping publishes records from that topic to a PubNub channel with the same name.

## Add your PubNub keys

Open `examples/pubnub-sink-connector.json`. It is the Kafka Connect configuration that creates the connector.

```json
{
    "name": "pubnub-sink-connector",
    "config": {
        "topics":"pubnub",
        "topics.regex":"",
        "pubnub.user_id" : "myUserId123",
        "pubnub.publish_key": "demo",
        "pubnub.subscribe_key": "demo",
        "pubnub.secret_key": "demo",
        "connector.class": "com.pubnub.kafka.connect.PubNubKafkaSinkConnector",
        "tasks.max": "3",
        "value.deserializer":"custom.class.serialization.JsonDeserializer",
        "value.serializer":"custom.class.serialization.JsonSerializer"

    }
}
```

Replace these three values:

* `pubnub.user_id` with `kafka-connect-tutorial`.
* `pubnub.publish_key` with the publish key from your keyset.
* `pubnub.subscribe_key` with the subscribe key from your keyset.

Leave `topics` set to `pubnub`. The included producer writes records to that topic, and the connector publishes them to the PubNub channel with the same name.

Leave `pubnub.secret_key` set to `demo`. This tutorial does not use Access Manager, so the connector does not need your keyset's secret key.

## Start Kafka and Kafka Connect

Start the Docker Compose project in the background:

```bash
docker compose up -d
```

Docker builds the connector, starts Kafka and Kafka Connect, and starts the sample producer. Wait until Kafka Connect responds:

```bash
curl --fail http://localhost:8083/connectors
```

The first response is an empty JSON array:

```json
[]
```

If the request cannot connect yet, wait a few seconds and run it again. Kafka Connect does not accept the connector configuration until the service has started.

## Deploy the connector

Send the configuration file to Kafka Connect:

```bash
curl --fail-with-body \
  --request POST \
  --header "Content-Type: application/json" \
  --data @examples/pubnub-sink-connector.json \
  http://localhost:8083/connectors
```

Kafka Connect returns JSON that includes the connector name `pubnub-sink-connector`. The sample producer is already writing records such as this to the `pubnub` topic:

```json
{ "timestamp": 1705689453 }
```

The connector publishes each record value as a PubNub message on the channel `pubnub`.

## Verify the message arrives

Open the [PubNub Debug Console](https://www.pubnub.com/docs/console/). Enter the publish key and subscribe key from the keyset you configured. Set the channel to `pubnub` and choose any User ID, such as `kafka-tutorial-observer`.

Messages begin appearing on the channel:

```json
{ "timestamp": 1705689453 }
```

The timestamp changes as the sample producer writes new Kafka records. Seeing those messages confirms the full path:

```text
Kafka topic pubnub → Kafka Connect → PubNub channel pubnub → Debug Console
```

## What happened

1. Docker Compose started a Kafka broker, Kafka Connect, the PubNub connector build, and a sample Kafka producer.
2. Kafka Connect loaded the PubNub Kafka Sink Connector with your keyset credentials.
3. The connector read records from the `pubnub` Kafka topic and published their values to the PubNub channel `pubnub`.
4. Debug Console subscribed to that channel and displayed the live messages.

## Stop the tutorial environment

Remove the connector, then stop the containers:

```bash
curl --fail --request DELETE http://localhost:8083/connectors/pubnub-sink-connector
docker compose down
```

## Next steps

* [Kafka integration overview](https://www.pubnub.com/docs/integrations/kafka-overview.md). Compare the Kafka Sink Connector with the PubNub-to-Kafka Events & Actions path.
* [Create a Kafka action](https://www.pubnub.com/docs/integrations/event-forwarding/create-kafka-action.md). Send matching PubNub events to a Kafka topic.
* [Receive messages](https://www.pubnub.com/docs/pub-sub/subscribe/receive-messages.md). Receive Kafka-originated PubNub messages in your application.

Last updated at: 2026-09-30T07:20:08.000Z