Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> SimpleConsumer - Should FetchRequests span log segments?


Copy link to this message
-
Re: SimpleConsumer - Should FetchRequests span log segments?
When you iterate the fetch result, MessageAndOffset has a nextOffet()
method. You need to use that offset to initiate the next fetch request. See
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Examplefor
an example.

Thanks,

Jun
On Fri, Jun 28, 2013 at 9:53 PM, Ran RanUser <[EMAIL PROTECTED]> wrote:

> Using Kafka 0.8, when specifying a starting offset and appropriate
> fetchSize, SimpleConsumer will only return up to the highest offset of the
> log segment containing the starting offset.
>
> For example,
>
> log segment #1 contains offsets 1 - 10
> log segment #2 contiains offsets 11 - 100
>
> A fetch request specifying offset 9, will only return offsets 9 and 10.
> Specifying starting offset 11 will return 11 - 100
>
> Must one check fetchResponse.highwatermark each time to determine if there
> are higher numbered offsets, and continue submitting fetchRequests ?
>
> I assumed SimpleConsumer would return offsets/messages spanning log
> segments, up to the specified fetchSize.
>
>
> Thanks!
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB