Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - [jira] [Updated] (KAFKA-716) SimpleConsumerPerformance does not consume all available messages


Copy link to this message
-
[jira] [Updated] (KAFKA-716) SimpleConsumerPerformance does not consume all available messages
"Neha Narkhede 2013-01-23, 18:55

     [ https://issues.apache.org/jira/browse/KAFKA-716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neha Narkhede updated KAFKA-716:
--------------------------------

    Labels: p2  (was: )
    
> SimpleConsumerPerformance does not consume all available messages
> -----------------------------------------------------------------
>
>                 Key: KAFKA-716
>                 URL: https://issues.apache.org/jira/browse/KAFKA-716
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: John Fung
>            Priority: Blocker
>              Labels: p2
>
> To reproduce the issue:
> 1. Start 1 zookeeper
> 2. Start 1 broker
> 3. Send some messages
> 4. Start SimpleConsumerPerformance to consume messages. The only way to consume all messages is to set the fetch-size to be greater than the log segment file size.
> 5. This output shows that SimpleConsumerPerformance consumes only 6 messages:
> $ bin/kafka-run-class.sh kafka.perf.SimpleConsumerPerformance --server kafka://host1:9092 --topic topic_001 --fetch-size 2048 --partition 0
> start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
> 2013-01-21 15:09:21:124, 2013-01-21 15:09:21:165, 2048, 0.0059, 0.1429, 6, 146.3415
> 6. This output shows that ConsoleConsumer consumes all 5500 messages (same test as the above)
> $ bin/kafka-run-class.sh kafka.consumer.ConsoleConsumer --zookeeper host2:2181 --topic topic_001 --consumer-timeout-ms 5000   --formatter kafka.consumer.ChecksumMessageFormatter  --from-beginning | grep ^checksum | wc -l
> Consumed 5500 messages
> 5500

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira