Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Question about resetting offsets and the high-level consumer


+
Bob Cotton 2012-11-09, 13:04
+
Jun Rao 2012-11-09, 15:30
+
Michal Haris 2012-11-10, 11:40
Copy link to this message
-
Re: Question about resetting offsets and the high-level consumer
Found it in archives, autooffset.reset is only used when the consumer
registering first time or the last committed offset is out-of-range, so
that means that we have to create direct connection to the zookeeper and
set/delete the /consumer/xyz/offset/topic123/*-* before starting the
consumer to get it reset to the head of the stream.. shouldn't be a big
problem to patch the consumer api with a method to do that right ?

On 10 November 2012 11:40, Michal Haris <[EMAIL PROTECTED]> wrote:

> Really? So what does the autooffset.reset=largest do ?
> On Nov 9, 2012 3:30 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
>
>> Bob,
>>
>> The high level consumer doesn't support resetting the starting offset
>> right
>> now. We will look into that post 0.8 as part of the consumer redesign.
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Fri, Nov 9, 2012 at 5:04 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:
>>
>> > We have a topic whose consumers, on startup, only need to consume from
>> the
>> > head of the topic. They don't need to consume old messages they may have
>> > missed.
>> >
>> > We would like to use the high-level consumer, as the API is a bit
>> simpler.
>> >
>> > Is there a way to reset the current offsets for the high-level consumer?
>> >
>> > We've though about deleting the consumer group nodes in ZK manually, but
>> > that seems brittle as those locations may change from release to
>> release.
>> >
>> > Is there this level of control somewhere I've missed?
>> >
>> > Thanks
>> > - Bob
>> >
>>
>
--
Michal Haris
Software Engineer

www.visualdna.com | t: +44 (0) 207 734 7033
+
Jun Rao 2012-11-12, 06:03
+
Michal Haris 2012-11-14, 00:08
+
Jun Rao 2012-11-14, 05:38
+
Michal Haris 2012-11-16, 00:22
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB