Kafkacat

Background

You can use any of the available clients to interface with your cluster. In this example we will use a command line producer and consumer called Kafkacat.

Prerequisites

Step 1: Get your Apache Kafka endpoint

  • From the deployments list - Click on Apache Kafka under the deployment you would like to send data to.
  • Click on the connections tab.
  • Copy the value for Plain Text Endpoint.
  • Open a terminal window and paste the endpoint value into an environment variable. This value should end in eventador.io:9092. It will look something like this:
bash
export BROKERS=sec-cust999303.eventador.io:9092

Step 2: Send some data

Now we’re ready to to send (produce) some data. In producer mode kafkacat reads messages from stdin and produces them to the provided Kafka cluster (-b) and topic (-t). For more information check the kafkacat man page.

First, let’s double check $BROKERS is set:

echo $BROKERS
sec-cust999303.eventador.io:9092  <---- should match value from plain text endpoint box

To send some data, let's send some simple JSON formatted text. Enter the below commands:

echo '{"sensor": 1, "temp": 100}' | kafkacat -P -b $BROKERS -t demo
echo '{"sensor": 2, "temp": 101}' | kafkacat -P -b $BROKERS -t demo

Step 3: Consume some data

This time we will use kafkacat to read (consume) from the topic. In consumer mode kafkacat reads messages from a topic and prints them to stdout.

echo $BROKERS
sec-cust999303.eventador.io:9092  <---- should match value from plain text endpoint box

Enter the below command:

kafkacat -C -b $BROKERS -t demo -o beginning -e -q
{"sensor": 2, "temp": 101}
{"sensor": 1, "temp": 100}

Step 4: Reading from the beginning offset

Let’s add a few more messages:

echo '{"sensor": 3, "temp": 95}' | kafkacat -P -b $BROKERS -t demo
echo '{"sensor": 4, "temp": 103}' | kafkacat -P -b $BROKERS -t demo

Now let’s read the topic from the beginning again:

kafkacat -C -b $BROKERS -t demo -o beginning -e -q
{"sensor": 3, "temp": 95}
{"sensor": 2, "temp": 101}
{"sensor": 1, "temp": 100}
{"sensor": 4, "temp": 103}

Note: Your messages may print in a different order. By default, producers rotate between available partitions. Records are guaranteed to be in order within a partition but when a consumer is subscribed to a topic with multiple partitions messages may be consumed in a different order than they were sent.