Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> SimpleConsumer - Should FetchRequests span log segments?


Copy link to this message
-
Re: SimpleConsumer - Should FetchRequests span log segments?
This is correct. It would be possible to change the server behavior to
avoid the smaller fetch responses but it would be more complex to
implement and there wouldn't be much benefit to this.

You can tell you are at the end of the log because either (1) you get
an empty message set back, or (2) the offset of the last message you
fetched is at the log end offset.

-Jay

On Fri, Jun 28, 2013 at 9:53 PM, Ran RanUser <[EMAIL PROTECTED]> wrote:
> Using Kafka 0.8, when specifying a starting offset and appropriate
> fetchSize, SimpleConsumer will only return up to the highest offset of the
> log segment containing the starting offset.
>
> For example,
>
> log segment #1 contains offsets 1 - 10
> log segment #2 contiains offsets 11 - 100
>
> A fetch request specifying offset 9, will only return offsets 9 and 10.
> Specifying starting offset 11 will return 11 - 100
>
> Must one check fetchResponse.highwatermark each time to determine if there
> are higher numbered offsets, and continue submitting fetchRequests ?
>
> I assumed SimpleConsumer would return offsets/messages spanning log
> segments, up to the specified fetchSize.
>
>
> Thanks!