Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Question about resetting offsets and the high-level consumer


+
Bob Cotton 2012-11-09, 13:04
+
Jun Rao 2012-11-09, 15:30
+
Michal Haris 2012-11-10, 11:40
+
Michal Haris 2012-11-10, 14:41
+
Jun Rao 2012-11-12, 06:03
Copy link to this message
-
Re: Question about resetting offsets and the high-level consumer
Michal Haris 2012-11-14, 00:08
Of course assuming that none of the consumers in the group are interested
in the missed messages in such case.
On Nov 12, 2012 6:03 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:

> It's easy if there is only a single consumer in the group. In general,
> however, there could be multiple consumers in the same group. So, some kind
> of coordination is needed.
>
> Thanks,
>
> Jun
>
> On Sat, Nov 10, 2012 at 6:41 AM, Michal Haris <[EMAIL PROTECTED]
> >wrote:
>
> > Found it in archives, autooffset.reset is only used when the consumer
> > registering first time or the last committed offset is out-of-range, so
> > that means that we have to create direct connection to the zookeeper and
> > set/delete the /consumer/xyz/offset/topic123/*-* before starting the
> > consumer to get it reset to the head of the stream.. shouldn't be a big
> > problem to patch the consumer api with a method to do that right ?
> >
> > On 10 November 2012 11:40, Michal Haris <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Really? So what does the autooffset.reset=largest do ?
> > > On Nov 9, 2012 3:30 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> > >
> > >> Bob,
> > >>
> > >> The high level consumer doesn't support resetting the starting offset
> > >> right
> > >> now. We will look into that post 0.8 as part of the consumer redesign.
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >>
> > >> On Fri, Nov 9, 2012 at 5:04 AM, Bob Cotton <[EMAIL PROTECTED]>
> > wrote:
> > >>
> > >> > We have a topic whose consumers, on startup, only need to consume
> from
> > >> the
> > >> > head of the topic. They don't need to consume old messages they may
> > have
> > >> > missed.
> > >> >
> > >> > We would like to use the high-level consumer, as the API is a bit
> > >> simpler.
> > >> >
> > >> > Is there a way to reset the current offsets for the high-level
> > consumer?
> > >> >
> > >> > We've though about deleting the consumer group nodes in ZK manually,
> > but
> > >> > that seems brittle as those locations may change from release to
> > >> release.
> > >> >
> > >> > Is there this level of control somewhere I've missed?
> > >> >
> > >> > Thanks
> > >> > - Bob
> > >> >
> > >>
> > >
> >
> >
> > --
> > Michal Haris
> > Software Engineer
> >
> > www.visualdna.com | t: +44 (0) 207 734 7033
> >
>
+
Jun Rao 2012-11-14, 05:38
+
Michal Haris 2012-11-16, 00:22