Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Question about resetting offsets and the high-level consumer


Copy link to this message
-
Re: Question about resetting offsets and the high-level consumer
Michal Haris 2012-11-16, 00:22
Well, resetting could follow the partition ownership where only owned
partitions can be reset by any one consumer. There's definitely a case for
consumer applications that are more about observing frequencies rather than
total aggregation and so are typically interested in last couple of seconds
/ minutes of the stream.

On 14 November 2012 05:38, Jun Rao <[EMAIL PROTECTED]> wrote:

> It's more than that. For example, what if 2 consumers want to set different
> offsets for the same partition?
>
> Jun
>
> On Tue, Nov 13, 2012 at 4:08 PM, Michal Haris <[EMAIL PROTECTED]
> >wrote:
>
> > Of course assuming that none of the consumers in the group are interested
> > in the missed messages in such case.
> > On Nov 12, 2012 6:03 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> >
> > > It's easy if there is only a single consumer in the group. In general,
> > > however, there could be multiple consumers in the same group. So, some
> > kind
> > > of coordination is needed.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sat, Nov 10, 2012 at 6:41 AM, Michal Haris <
> > [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Found it in archives, autooffset.reset is only used when the consumer
> > > > registering first time or the last committed offset is out-of-range,
> so
> > > > that means that we have to create direct connection to the zookeeper
> > and
> > > > set/delete the /consumer/xyz/offset/topic123/*-* before starting the
> > > > consumer to get it reset to the head of the stream.. shouldn't be a
> big
> > > > problem to patch the consumer api with a method to do that right ?
> > > >
> > > > On 10 November 2012 11:40, Michal Haris <[EMAIL PROTECTED]>
> > > > wrote:
> > > >
> > > > > Really? So what does the autooffset.reset=largest do ?
> > > > > On Nov 9, 2012 3:30 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > >> Bob,
> > > > >>
> > > > >> The high level consumer doesn't support resetting the starting
> > offset
> > > > >> right
> > > > >> now. We will look into that post 0.8 as part of the consumer
> > redesign.
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Jun
> > > > >>
> > > > >>
> > > > >> On Fri, Nov 9, 2012 at 5:04 AM, Bob Cotton <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >>
> > > > >> > We have a topic whose consumers, on startup, only need to
> consume
> > > from
> > > > >> the
> > > > >> > head of the topic. They don't need to consume old messages they
> > may
> > > > have
> > > > >> > missed.
> > > > >> >
> > > > >> > We would like to use the high-level consumer, as the API is a
> bit
> > > > >> simpler.
> > > > >> >
> > > > >> > Is there a way to reset the current offsets for the high-level
> > > > consumer?
> > > > >> >
> > > > >> > We've though about deleting the consumer group nodes in ZK
> > manually,
> > > > but
> > > > >> > that seems brittle as those locations may change from release to
> > > > >> release.
> > > > >> >
> > > > >> > Is there this level of control somewhere I've missed?
> > > > >> >
> > > > >> > Thanks
> > > > >> > - Bob
> > > > >> >
> > > > >>
> > > > >
> > > >
> > > >
> > > > --
> > > > Michal Haris
> > > > Software Engineer
> > > >
> > > > www.visualdna.com | t: +44 (0) 207 734 7033
> > > >
> > >
> >
>

--
Michal Haris
Software Engineer

www.visualdna.com | t: +44 (0) 207 734 7033