Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> [jira] [Commented] (KAFKA-761) Fetch request randomly returns 0 bytes when there's still data in the log


Copy link to this message
-
[jira] [Commented] (KAFKA-761) Fetch request randomly returns 0 bytes when there's still data in the log

    [ https://issues.apache.org/jira/browse/KAFKA-761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597377#comment-13597377 ]

Lorenzo Alberton commented on KAFKA-761:
----------------------------------------

Some more details: this happens with a moderately high number of topics (a few hundreds), when the consumers connect and disconnect quite often to fetch a few batches at a time.
                
> Fetch request randomly returns 0 bytes when there's still data in the log
> -------------------------------------------------------------------------
>
>                 Key: KAFKA-761
>                 URL: https://issues.apache.org/jira/browse/KAFKA-761
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.7.2
>         Environment: centos 6
>            Reporter: Lorenzo Alberton
>            Assignee: Neha Narkhede
>
> I can't track the bug down in the code, but the behaviour has been hitting us for quite some time (on Kafka 0.7.2): sometimes, a new Simple Consumer object, on a new Fetch Request, returns an empty byte buffer (or a number of bytes considerably smaller than the buffer size), even if there's more data on disk.
> This happens randomly, with no errors, at different offsets in the same topic. Different consumer groups processing the same topic might get stuck at different offsets.
> There seems to be no correlation between empty responses and the offset, or the buffer size, or the log file size. At first we thought it only happened when reaching the end of a log file, before jumping to the next file, but it doesn't seem to be the case. It often happens in the middle of a file.
> Simply retrying the Fetch Request multiple times works quite reliably, after 2 or 3 empty fetches the consumer eventually gets unstuck (we check the size of the queue to see if there's more data before retrying).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB