Benchmarking Apache Kafka using DI-KafkaMeter

Introduction:

Apache Kafka is an open-source distributed event streaming platform which provides publish and subscribe methodology to Read/Write, Store and Process the real time events.

It was developed by LinkedIn initially and it is built for real-time data ingestion, data integration and messaging at a larger scale. It works admirably well with huge amounts of data arriving at high velocity, allowing to achieve high throughput, high durability and high availability. This is why it secured a great place in Bigdata. Where it plays a key role in processing huge data and event streams at the heart of their architecture.

Use Cases:

Apache Kafka can be used in various use cases, where it fits perfectly for huge data processing. Some of the high use cases were,

  • Messaging
  • Activity tracking
  • Metrics
  • Log Aggregation
  • Stream Processing
  • Event sourcing, etc

Key Terms:

Most of the Kafka operations are within the below terms.

  • Producers — Client which is sending the events/messages to Kafka broker to one or more topic
  • Consumers — Client who reads the events/messages from Kafka topic
  • Topics — It is a category/group/directory to store events/messages produced by the producers. A topic can have zero, one, or many producers that write events to it, as well as zero, one, or many consumers that subscribe to these events.
  • Zookeeper — It is to manage the Kafka nodes in the cluster. Its main operations are
    - Controller Election
    - Cluster Membership
    - Topic configuration
    - Access controls
    - Quotas

Load Test Using DI-KafkaMeter:

In Realtime, Kafka will be processing huge amount of events generated from multiple applications. One of the key area where Kafka is highly used is at BigData. To reproduce the event generation to Kafka topics, we need to pump a huge load from a tool.

Apache JMeter provides enduser the capability to extend its feature by adding custom plugin, which will help us to focus on monitoring and tuning the parameters for efficient results. DI-KafkaMeter is one such plugin that can help us to load test Kafka easily using JMeter. There were other plugins which are available in JMeter are pepper-box, kafkameter. I prefer DI-Kafkameter for its simplicity and handy to use structure.

Setting up Producer config:

Once you download the DI-KafkaMeter from Plugins manager. You will be able to see a config element and sampler added specifically to load test Apache Kafka. You need to Add the KafkaProducerConfig to your test plan (Add-> Config -> KafkaProducerConfig). It enabled the JMeter to establish a connection with Kafka Brokers with the required configurations.

Kafka Producer Config from DI-KafkaMeter

To create a connection with the Kafka brokers you need the below,

- Bootstrap server/Kafka brokers (Mandatory)Addresses of brokers/ zookeepers in the format of broker-ip-1: port, broker-ip-2: port, etc.

- Client ID (Mandatory)Unique Id for the client, from where you are sending the event (in our case, it’s hostname of JMeter)

- Batch size (Optional)This is the largest message size.

- Serializer key (Optional)Key for serialization/deserialization. Leave it to default if you do not have explicit serialization

- Additional configs (Optional)– Area to add more configs, if required

- SSL Configs (Optional)– If you use SSL to connect to brokers

Setting up Producer Sampler:

Sampler enables the JMeter to publish the messages to a topic from the acquired connection using the Config element. The Kafka sampler can only fit under a ThreadGroup (Threadgroup -> Sampler-> KafkaProducerSampler).

Kafka Producer Sampler from DI-KafkaMeter

To Send a message to Kafka brokers you need the below,

- Kafka Topic (Mandatory) — Topic name, where you want to send messages

- Kafka message (Mandatory)– Original message you want to send

- Partition string (Optional)– You can send message to a particular partition within Kafka.

- Kafka message key (Optional)– key for your message to fetch it

Running Load Test:

Once the above configuration is successful, you can use JMeter to run a load test on Kafka Brokers to evaluate/benchmark the performance. The testing process remains same.

Conclusion:

I personally believes that Apache JMeter has more capabilities than what it looks like. All you need is, a right - handy plugin to evaluate the performance of any application.

I have contributed the DI-KafkaMeter to the JMeter community, which is now available in JMeter’s plugin manager. Also, If you like my work, Hit a star to my Github Repo

Will come up with another unique article. Keep supporting, Click on follow button and hit the subscribe to get the notification on new article publish.

Stay positive, Happy engineering !

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store