Create Kafka action

To send PubNub data into Kafka, you must first configure your Kafka environment and the Kafka action in Admin Portal section.

For Events & Actions, it doesn't matter how or where you host your Kafka instance as long as you correctly configure your action in Admin Portal.

Receive Kafka events in PubNub

Apart from streaming PubNub events to Kafka through Kafka Action, PubNub lets you receive Kafka events in PubNub. For more details, read about PubNub Kafka Sink Connector.

Existing Kafka cluster

You will need the following information to configure Events & Actions to connect to an Apache Kafka event streaming platform.

  • Topic and key
  • Broker URL and port
  • Username and password

When you have all this information, go to the Configure Admin Portal section.

New Kafka cluster

There are many ways to create and host Kafka clusters. The Kafka action within Events & Actions is technology-agnostic. Regardless of how you host your instance, the Kafka action will connect.

If you need guidance on creating a new Kafka cluster, read on.

Refer to Kafka Quick Start documentation to understand how to create a local Kafka cluster. Ensure to have the following information ready as you will need it to create a Kafka action:

  • Topic and key
  • Broker URL and port
  • Username and password

When you've created your Kafka cluster and have access to this information, create a Kafka action by following the steps described in the Configure Admin Portal section.

Configure Admin Portal

  1. In the Events & Actions view on the Admin Portal, create an action by clicking on the + Add Action button.

  2. Select Apache Kafka as the action type.

  3. In the Routing Key field, type the name of the topic you created previously, for example, topic_0.

  4. Still in the Routing Key field, type your key after the topic in the following format: topic:key, for example, topic_0:stamford.

    Key and message order

    Each message loaded into a topic may have a key. Kafka, being a distributed system, divides topics into partitions, and message order is guaranteed only within the same partition. Adding a key to the topic name ensures the messages are sent to the same partition, preserving the correct order.

  5. In the Authentication Mechanism drop-down, select the authentication type used by your Kafka setup. Supported authentication types include SASL/PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512.

  6. In the Username field, type your Kafka username. In Confluent Cloud, this is the value of the Key credential you noted while creating an API key.

    Unsecured connections

    PubNub doesn't allow unsecured connections to Kafka.

  7. In the Password field, type your Kafka password. In Confluent Cloud, this is the value of the Secret credential you noted while creating an API key.

  8. In the Brokers field, provide a URL of the Kafka cluster to send the events to, for example, 574mf0rd-bridge.ldn-west:1905. In Confluent Cloud, this is the value of the Bootstrap server parameter you noted while configuring a Kafka cluster.

    Broker list

    You can provide a list of URLs in the following format: hostname:port, hostname2:port.

  9. Optionally, enable and configure retries through the Kafka retry option.

  10. Pair your action with an event listener without leaving the Actions view. To do this, click the Add event listener button and select an existing event listener or create a new one.

  11. Save your newly created action by clicking the Save changes button.

Last updated on