Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Question about resetting offsets and the high-level consumer


Copy link to this message
-
Re: Question about resetting offsets and the high-level consumer
It's more than that. For example, what if 2 consumers want to set different
offsets for the same partition?

Jun

On Tue, Nov 13, 2012 at 4:08 PM, Michal Haris <[EMAIL PROTECTED]>wrote:

> Of course assuming that none of the consumers in the group are interested
> in the missed messages in such case.
> On Nov 12, 2012 6:03 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
>
> > It's easy if there is only a single consumer in the group. In general,
> > however, there could be multiple consumers in the same group. So, some
> kind
> > of coordination is needed.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sat, Nov 10, 2012 at 6:41 AM, Michal Haris <
> [EMAIL PROTECTED]
> > >wrote:
> >
> > > Found it in archives, autooffset.reset is only used when the consumer
> > > registering first time or the last committed offset is out-of-range, so
> > > that means that we have to create direct connection to the zookeeper
> and
> > > set/delete the /consumer/xyz/offset/topic123/*-* before starting the
> > > consumer to get it reset to the head of the stream.. shouldn't be a big
> > > problem to patch the consumer api with a method to do that right ?
> > >
> > > On 10 November 2012 11:40, Michal Haris <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Really? So what does the autooffset.reset=largest do ?
> > > > On Nov 9, 2012 3:30 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> > > >
> > > >> Bob,
> > > >>
> > > >> The high level consumer doesn't support resetting the starting
> offset
> > > >> right
> > > >> now. We will look into that post 0.8 as part of the consumer
> redesign.
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Jun
> > > >>
> > > >>
> > > >> On Fri, Nov 9, 2012 at 5:04 AM, Bob Cotton <[EMAIL PROTECTED]>
> > > wrote:
> > > >>
> > > >> > We have a topic whose consumers, on startup, only need to consume
> > from
> > > >> the
> > > >> > head of the topic. They don't need to consume old messages they
> may
> > > have
> > > >> > missed.
> > > >> >
> > > >> > We would like to use the high-level consumer, as the API is a bit
> > > >> simpler.
> > > >> >
> > > >> > Is there a way to reset the current offsets for the high-level
> > > consumer?
> > > >> >
> > > >> > We've though about deleting the consumer group nodes in ZK
> manually,
> > > but
> > > >> > that seems brittle as those locations may change from release to
> > > >> release.
> > > >> >
> > > >> > Is there this level of control somewhere I've missed?
> > > >> >
> > > >> > Thanks
> > > >> > - Bob
> > > >> >
> > > >>
> > > >
> > >
> > >
> > > --
> > > Michal Haris
> > > Software Engineer
> > >
> > > www.visualdna.com | t: +44 (0) 207 734 7033
> > >
> >
>