Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Question about resetting offsets and the high-level consumer


+
Bob Cotton 2012-11-09, 13:04
+
Jun Rao 2012-11-09, 15:30
+
Michal Haris 2012-11-10, 11:40
+
Michal Haris 2012-11-10, 14:41
Copy link to this message
-
Re: Question about resetting offsets and the high-level consumer
It's easy if there is only a single consumer in the group. In general,
however, there could be multiple consumers in the same group. So, some kind
of coordination is needed.

Thanks,

Jun

On Sat, Nov 10, 2012 at 6:41 AM, Michal Haris <[EMAIL PROTECTED]>wrote:

> Found it in archives, autooffset.reset is only used when the consumer
> registering first time or the last committed offset is out-of-range, so
> that means that we have to create direct connection to the zookeeper and
> set/delete the /consumer/xyz/offset/topic123/*-* before starting the
> consumer to get it reset to the head of the stream.. shouldn't be a big
> problem to patch the consumer api with a method to do that right ?
>
> On 10 November 2012 11:40, Michal Haris <[EMAIL PROTECTED]>
> wrote:
>
> > Really? So what does the autooffset.reset=largest do ?
> > On Nov 9, 2012 3:30 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> >
> >> Bob,
> >>
> >> The high level consumer doesn't support resetting the starting offset
> >> right
> >> now. We will look into that post 0.8 as part of the consumer redesign.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Fri, Nov 9, 2012 at 5:04 AM, Bob Cotton <[EMAIL PROTECTED]>
> wrote:
> >>
> >> > We have a topic whose consumers, on startup, only need to consume from
> >> the
> >> > head of the topic. They don't need to consume old messages they may
> have
> >> > missed.
> >> >
> >> > We would like to use the high-level consumer, as the API is a bit
> >> simpler.
> >> >
> >> > Is there a way to reset the current offsets for the high-level
> consumer?
> >> >
> >> > We've though about deleting the consumer group nodes in ZK manually,
> but
> >> > that seems brittle as those locations may change from release to
> >> release.
> >> >
> >> > Is there this level of control somewhere I've missed?
> >> >
> >> > Thanks
> >> > - Bob
> >> >
> >>
> >
>
>
> --
> Michal Haris
> Software Engineer
>
> www.visualdna.com | t: +44 (0) 207 734 7033
>
+
Michal Haris 2012-11-14, 00:08
+
Jun Rao 2012-11-14, 05:38
+
Michal Haris 2012-11-16, 00:22
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB