Create Kafka action

To send PubNub data to Kafka, first configure your Kafka environment and then create a Kafka Action in the Admin Portal.

For Events & Actions, hosting is agnostic. Configure the Kafka Action in the Admin Portal to connect to any supported cluster.

Receive Kafka events in PubNub

Besides streaming PubNub events to Kafka using a Kafka Action, you can also receive Kafka events in PubNub. See the PubNub Kafka Sink Connector.

Existing Kafka cluster

You need the following information to connect Events & Actions to Apache Kafka:

• Topic and key
• Broker URL and port
• Username and password

Then go to the Configure Admin Portal section.

New Kafka cluster

There are many ways to create and host Kafka clusters. The Kafka Action in Events & Actions is technology‑agnostic. Regardless of where you host Kafka, the Kafka Action can connect.

If you need guidance on creating a cluster, read on.

Self-hosted Kafka

Use the Kafka Quick Start to create a local Kafka cluster. Have this information ready to create a Kafka Action:

• Topic and key
• Broker URL and port
• Username and password

After you create the cluster and gather this information, follow Configure Admin Portal.

Amazon MSK cluster using Terraform

Use the Terraform guide to create an Amazon MSK cluster. Have this information ready to create a Kafka Action:

• Topic and key
• Broker URL and port
• Username and password

After you create the cluster and gather this information, follow Configure Admin Portal.

Confluent Cloud

This procedure uses the Confluent Cloud UI to set up your Kafka environment. It has three parts:

1. Configure a Kafka cluster.
2. Create a topic.
3. Create an API key.

For more on creating clusters on Confluent Cloud, see Quick Start for Confluent Cloud.

Configure a Kafka cluster

Create a cluster first. Then:

1. Log in to Confluent Cloud and create a Kafka cluster to receive PubNub messages.
2. Select the cloud provider and cluster type. Complete the wizard.
3. After the cluster launches, open Cluster Settings and note the Bootstrap server. You will need it in the Admin Portal.

Create a Kafka topic

Create a Kafka topic before sending PubNub data. Creating the topic first ensures delivery.

To create a topic:

1. On the cluster overview page in Confluent Cloud, click Topics.
2. On the Topics page, click Create topic and configure it.
3. After the topic is created, note its name. You will need it in the Admin Portal.

Create API key

Use secure connections by setting up Kafka credentials.

To create an API key:

1. On the cluster overview page in Confluent Cloud, click API keys. The API keys page opens.
2. On the API keys page, click Create key and configure it.

   Secret visibility

   Do not click Download and continue yet. Doing so hides the Secret credential.

3. Note the Key and Secret values. You will need them in the Admin Portal.
4. Click Download and continue to finish creating the key.

Configure Admin Portal

1. In the Events & Actions view on the Admin Portal, click + Add Action.
2. Select Apache Kafka as the action type.
3. In Routing Key, enter your topic, for example, topic_0.
4. To add a key, enter topic:key, for example, topic_0:stamford.

   Key and message order

   Each message sent to a topic may include a key. Kafka partitions topics and guarantees order only within the same partition. Adding a key ensures messages go to the same partition and preserves order.

5. In Authentication Mechanism, select the type your Kafka setup uses. Supported types include SASL/PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512.
6. In Username, enter your Kafka username. In Confluent Cloud, this is the Key you noted when you created an API key.

   Secure connections

   Use secure connections. PubNub blocks unsecured connections to Kafka.

7. In Password, enter your Kafka password. In Confluent Cloud, this is the Secret you noted when you created an API key.
8. In Brokers, enter the cluster URL, for example, 574mf0rd-bridge.ldn-west:1905. In Confluent Cloud, use the Bootstrap server from Configure a Kafka cluster.

   Broker list

   You can provide a list in this format: hostname:port, hostname2:port.

9. (Optional) Enable and configure retries in Kafka retry.
10. Pair the action with an event listener. Click Add event listener and select an existing listener or create one. Link it to a listener that captures the events you want.
11. Click Save changes.