Kafka integration overview
PubNub and Apache Kafka connect in two separate directions. The PubNub Kafka Sink Connector publishes Kafka records to PubNub channels. An Events & Actions Kafka Action forwards matching PubNub events to Kafka. They solve opposite data paths and use separate configuration.
Choose a direction
| If you need to | Use | It runs in |
|---|---|---|
| Send records from Kafka topics to connected PubNub clients | PubNub Kafka Sink Connector | Your Kafka Connect environment |
| Forward PubNub events to a Kafka topic | Events & Actions Kafka Action | PubNub, configured in the Admin Portal |
The Kafka Sink Connector is the inbound path to PubNub. A Kafka Action is the outbound path from PubNub. They are complementary, but they are not one bidirectional connector.
How the paths work
On the inbound path, a backend producer writes records to a Kafka topic. The PubNub Kafka Sink Connector runs in Kafka Connect, reads those records, and publishes each one to a PubNub channel. PubNub then delivers each message to the client apps subscribed to that channel.
On the outbound path, activity in a client app creates a PubNub event. An Events & Actions event listener matches the event, and its Kafka Action forwards the event data to a Kafka topic.
Send Kafka records to PubNub
The PubNub Kafka Sink Connector runs as a Kafka Connect plugin in your Kafka environment. By default, it publishes each Kafka record's value to a PubNub channel with the same name as the record's topic.
That default mapping keeps a simple integration easy to reason about. A producer writes to the Kafka topic orders, the connector publishes its records to the PubNub channel orders, and clients subscribe to that channel. You can provide a custom router when your application needs different channel names or message payloads.
The connector holds your PubNub credentials in server-side Kafka Connect configuration. It needs a secret key only if you use it to grant Access Manager tokens. Otherwise, keep your publish and subscribe keys in that configuration and out of client applications and source control.
Forward PubNub events to Kafka
Events & Actions forwards PubNub activity to external systems without SDK code. An event listener watches events on a keyset. When an event matches, its Kafka Action sends the event data to your Kafka topic.
This path works when Kafka needs a copy of activity that already occurred in PubNub, such as messages, Presence changes, or App Context updates. It does not deploy or manage Kafka Connect. You configure the listener and Kafka Action in the Admin Portal.
The Kafka destination receives the event data selected by the listener. Treat Kafka as a separate system for access control, retention, and privacy decisions.
Next steps
- Stream Kafka records to PubNub clients. Run the Kafka Sink Connector locally and verify that a Kafka record reaches a PubNub channel.
- Create a Kafka action. Forward matching PubNub events to a Kafka topic.
- Events & Actions. Understand event listeners, actions, retries, and billing.