Fully Managed Apache Kafka

Eventador Fully Managed Apache Kafka is a simple, secure and fully managed Apache Kafka platform.

You can run Apache Kafka as an Instant-On (public) plan or an enterprise plan (your account). To run it in an enterprise plan follow these steps to get set up:

  • Contact sales for a trial account and/or demo. They will give you a dedicated URL and environment for your use.
  • Login to your specific URL given to you by sales/support to create ESP deployments.

Create a Deployment

A deployment is all the components needed to run Apache Kafka:

  1. Dedicated Zookeeper cluster
  2. Dedicated Apache Kafka cluster
  3. Network access and routing rules
  4. Monitoring and stats gathering agents
  5. Access control manger
  6. Unique VPC.
  7. Brokers are spread across AWS availability zones.
  8. VIP's for Kafka endpoints.

To create a deployment:

  • Click the Deployments tab from the Eventador Console.
  • Click Create Deployment.
  • Select a plan.
  • Select an AWS region.
  • Name the deployment.
  • Click Create.

Screenshot

Create a Topic

A topic is a container to hold a stream of related data in Apache Kafka.

  • From the deployments list - Click the Apache Kafka link in your deployment.
  • Click on Add Topic.
  • Name the topic demo and set the number of partitions to 16. Replication factor is set to 1 for development cluster but can be increased in production deployments.
  • Click Create.

Congratulations! You have setup a new Apache Kafka cluster. Now that your topic is created you’re ready to start sending data. Here is an example of producing and consuming data from your Apache Kafka deployment with Kafkacat.

screenshot

Delete a Deployment

  • From the deployments list - Click the Apache Kafka link in your deployment.
  • Click Settings.
  • Type the name of the deployment into the Delete Deployment box.
  • Click Delete.

NOTE: This operation can not be undone and is permanent.

Scaling Kafka

Clusters are scaled by adding brokers of the same instance type as the other brokers. Additional brokers are assigned an AWS availability zone based on a simple round robin algorithm.

  • From the deployments list - Click the Apache Kafka link in your deployment.
  • Click the Brokers tab.
  • Click the Add Brokers button.
  • Input the new number of brokers you would like. Currently only scaling UP is supported.
  • Click Add. The brokers will be added in the background.

NOTE: This is an online operation. Existing topics will not be rebalanced across new brokers. If you would like to rebalance topics, contact Support. New topics will automatically be partitioned across the new brokers.

Kafka Endpoint

Once a deployment is created you have a simple endpoint connection (VIP:port) to use in your applications and when connecting to Kafka. To get the endpoint to use in your applications:

  • From the deployments list - Click on your deployment name - in this case PrototypeCluster.
  • Click on the Apache Kafka link.
  • Click on the Connections tab.
  • Copy+Paste the Plain Text Endpoint value into your applications.
  • Optionally, SSL/SASL endpoints are also exposed. There are code samples on how to use them with popular languages.

NOTE: We highly recommend using SSL connections.

Zookeeper Endpoint

Eventador is a fully managed service so normally there is no need to worry about zookeeper at all. In the event some tooling needs to use Zookeeper to connect - you have access and can specify ZK nodes like:

  • From the deployments list - Click on your deployment name - in this case PrototypeCluster.
  • Click on the Apache Kafka link.
  • Click on the Connections tab.
  • Replace the string kafka in the Plain Text Endpoint with zookeeper. Essentially: s/kafka/zookeeper/g
  • Use this connect string in tooling that requires a Zookeeper connection.