Benchmarking Apache Kafka using DI-KafkaMeter
Apache Kafka is an open-source distributed event streaming platform which provides publish and subscribe methodology to Read/Write, Store and Process the real time events.
It was developed by LinkedIn initially and it is built for real-time data ingestion, data integration and messaging at a larger scale. It works admirably well with huge amounts of data arriving at high velocity, allowing to achieve high throughput, high durability and high availability. This is why it secured a great place in Bigdata. Where it plays a key role in processing huge data and event streams at the heart of their architecture.
Apache Kafka can be used in various use cases, where it fits perfectly for huge data processing. Some of the high use cases were,
- Activity tracking
- Log Aggregation
- Stream Processing
- Event sourcing, etc
Most of the Kafka operations are within the below terms.
- Producers — Client which is sending the events/messages to Kafka broker to one or more topic
- Consumers — Client who reads the events/messages from Kafka topic
- Topics — It is a category/group/directory to store events/messages produced by the producers. A topic can have zero, one, or many producers that write events to it, as well as zero, one, or many consumers that subscribe to these events.
- Zookeeper — It is to manage the Kafka nodes in the cluster. Its main operations are
- Controller Election
- Cluster Membership
- Topic configuration
- Access controls
Load Test Using DI-KafkaMeter:
In Realtime, Kafka will be processing huge amount of events generated from multiple applications. One of the key area where Kafka is highly used is at BigData. To reproduce the event generation to Kafka topics, we need to pump a huge load from a tool.
Apache JMeter provides enduser the capability to extend its feature by adding custom plugin, which will help us to focus on monitoring and tuning the parameters for efficient results. DI-KafkaMeter is one such plugin that can help us to load test Kafka easily using JMeter. There were other plugins which are available in JMeter are pepper-box, kafkameter. I prefer DI-Kafkameter for its simplicity and handy to use structure.
Setting up Producer config:
Once you download the DI-KafkaMeter from Plugins manager. You will be able to see a config element and sampler added specifically to load test Apache Kafka. You need to Add the KafkaProducerConfig to your test plan (Add-> Config -> KafkaProducerConfig). It enabled the JMeter to establish a connection with Kafka Brokers with the required configurations.
To create a connection with the Kafka brokers you need the below,
- Bootstrap server/Kafka brokers (Mandatory)– Addresses of brokers/ zookeepers in the format of broker-ip-1: port, broker-ip-2: port, etc.
- Client ID (Mandatory)– Unique Id for the client, from where you are sending the event (in our case, it’s hostname of JMeter)
- Batch size (Optional)– This is the largest message size.
- Serializer key (Optional)– Key for serialization/deserialization. Leave it to default if you do not have explicit serialization
- Additional configs (Optional)– Area to add more configs, if required
- SSL Configs (Optional)– If you use SSL to connect to brokers
Setting up Producer Sampler:
Sampler enables the JMeter to publish the messages to a topic from the acquired connection using the Config element. The Kafka sampler can only fit under a ThreadGroup (Threadgroup -> Sampler-> KafkaProducerSampler).
To Send a message to Kafka brokers you need the below,
- Kafka Topic (Mandatory) — Topic name, where you want to send messages
- Kafka message (Mandatory)– Original message you want to send
- Partition string (Optional)– You can send message to a particular partition within Kafka.
- Kafka message key (Optional)– key for your message to fetch it
Running Load Test:
Once the above configuration is successful, you can use JMeter to run a load test on Kafka Brokers to evaluate/benchmark the performance. The testing process remains same.
I personally believes that Apache JMeter has more capabilities than what it looks like. All you need is, a right - handy plugin to evaluate the performance of any application.
Will come up with another unique article. Keep supporting, Click on follow button and hit the subscribe to get the notification on new article publish.
Stay positive, Happy engineering !