We have a topic whose consumers, on startup, only need to consume from the
head of the topic. They don't need to consume old messages they may have
missed.

We would like to use the high-level consumer, as the API is a bit simpler.

Is there a way to reset the current offsets for the high-level consumer?

We've though about deleting the consumer group nodes in ZK manually, but
that seems brittle as those locations may change from release to release.

Is there this level of control somewhere I've missed?

Thanks
- Bob

Bob,

The high level consumer doesn't support resetting the starting offset right
now. We will look into that post 0.8 as part of the consumer redesign.

Thanks,

Jun
On Fri, Nov 9, 2012 at 5:04 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:

> We have a topic whose consumers, on startup, only need to consume from the
> head of the topic. They don't need to consume old messages they may have
> missed.
>
> We would like to use the high-level consumer, as the API is a bit simpler.
>
> Is there a way to reset the current offsets for the high-level consumer?
>
> We've though about deleting the consumer group nodes in ZK manually, but
> that seems brittle as those locations may change from release to release.
>
> Is there this level of control somewhere I've missed?
>
> Thanks
> - Bob
>

Really? So what does the autooffset.reset=largest do ?
On Nov 9, 2012 3:30 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:

> Bob,
>
> The high level consumer doesn't support resetting the starting offset right
> now. We will look into that post 0.8 as part of the consumer redesign.
>
> Thanks,
>
> Jun
>
>
> On Fri, Nov 9, 2012 at 5:04 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:
>
> > We have a topic whose consumers, on startup, only need to consume from
> the
> > head of the topic. They don't need to consume old messages they may have
> > missed.
> >
> > We would like to use the high-level consumer, as the API is a bit
> simpler.
> >
> > Is there a way to reset the current offsets for the high-level consumer?
> >
> > We've though about deleting the consumer group nodes in ZK manually, but
> > that seems brittle as those locations may change from release to release.
> >
> > Is there this level of control somewhere I've missed?
> >
> > Thanks
> > - Bob
> >
>

Found it in archives, autooffset.reset is only used when the consumer
registering first time or the last committed offset is out-of-range, so
that means that we have to create direct connection to the zookeeper and
set/delete the /consumer/xyz/offset/topic123/*-* before starting the
consumer to get it reset to the head of the stream.. shouldn't be a big
problem to patch the consumer api with a method to do that right ?

On 10 November 2012 11:40, Michal Haris <[EMAIL PROTECTED]> wrote:

> Really? So what does the autooffset.reset=largest do ?
> On Nov 9, 2012 3:30 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
>
>> Bob,
>>
>> The high level consumer doesn't support resetting the starting offset
>> right
>> now. We will look into that post 0.8 as part of the consumer redesign.
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Fri, Nov 9, 2012 at 5:04 AM, Bob Cotton <[EMAIL PROTECTED]> wrote:
>>
>> > We have a topic whose consumers, on startup, only need to consume from
>> the
>> > head of the topic. They don't need to consume old messages they may have
>> > missed.
>> >
>> > We would like to use the high-level consumer, as the API is a bit
>> simpler.
>> >
>> > Is there a way to reset the current offsets for the high-level consumer?
>> >
>> > We've though about deleting the consumer group nodes in ZK manually, but
>> > that seems brittle as those locations may change from release to
>> release.
>> >
>> > Is there this level of control somewhere I've missed?
>> >
>> > Thanks
>> > - Bob
>> >
>>
>
--
Michal Haris
Software Engineer

www.visualdna.com | t: +44 (0) 207 734 7033

It's easy if there is only a single consumer in the group. In general,
however, there could be multiple consumers in the same group. So, some kind
of coordination is needed.

Thanks,

Jun

On Sat, Nov 10, 2012 at 6:41 AM, Michal Haris <[EMAIL PROTECTED]>wrote:

> Found it in archives, autooffset.reset is only used when the consumer
> registering first time or the last committed offset is out-of-range, so
> that means that we have to create direct connection to the zookeeper and
> set/delete the /consumer/xyz/offset/topic123/*-* before starting the
> consumer to get it reset to the head of the stream.. shouldn't be a big
> problem to patch the consumer api with a method to do that right ?
>
> On 10 November 2012 11:40, Michal Haris <[EMAIL PROTECTED]>
> wrote:
>
> > Really? So what does the autooffset.reset=largest do ?
> > On Nov 9, 2012 3:30 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> >
> >> Bob,
> >>
> >> The high level consumer doesn't support resetting the starting offset
> >> right
> >> now. We will look into that post 0.8 as part of the consumer redesign.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Fri, Nov 9, 2012 at 5:04 AM, Bob Cotton <[EMAIL PROTECTED]>
> wrote:
> >>
> >> > We have a topic whose consumers, on startup, only need to consume from
> >> the
> >> > head of the topic. They don't need to consume old messages they may
> have
> >> > missed.
> >> >
> >> > We would like to use the high-level consumer, as the API is a bit
> >> simpler.
> >> >
> >> > Is there a way to reset the current offsets for the high-level
> consumer?
> >> >
> >> > We've though about deleting the consumer group nodes in ZK manually,
> but
> >> > that seems brittle as those locations may change from release to
> >> release.
> >> >
> >> > Is there this level of control somewhere I've missed?
> >> >
> >> > Thanks
> >> > - Bob
> >> >
> >>
> >
>
>
> --
> Michal Haris
> Software Engineer
>
> www.visualdna.com | t: +44 (0) 207 734 7033
>

Of course assuming that none of the consumers in the group are interested
in the missed messages in such case.
On Nov 12, 2012 6:03 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:

> It's easy if there is only a single consumer in the group. In general,
> however, there could be multiple consumers in the same group. So, some kind
> of coordination is needed.
>
> Thanks,
>
> Jun
>
> On Sat, Nov 10, 2012 at 6:41 AM, Michal Haris <[EMAIL PROTECTED]
> >wrote:
>
> > Found it in archives, autooffset.reset is only used when the consumer
> > registering first time or the last committed offset is out-of-range, so
> > that means that we have to create direct connection to the zookeeper and
> > set/delete the /consumer/xyz/offset/topic123/*-* before starting the
> > consumer to get it reset to the head of the stream.. shouldn't be a big
> > problem to patch the consumer api with a method to do that right ?
> >
> > On 10 November 2012 11:40, Michal Haris <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Really? So what does the autooffset.reset=largest do ?
> > > On Nov 9, 2012 3:30 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> > >
> > >> Bob,
> > >>
> > >> The high level consumer doesn't support resetting the starting offset
> > >> right
> > >> now. We will look into that post 0.8 as part of the consumer redesign.
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >>
> > >> On Fri, Nov 9, 2012 at 5:04 AM, Bob Cotton <[EMAIL PROTECTED]>
> > wrote:
> > >>
> > >> > We have a topic whose consumers, on startup, only need to consume
> from
> > >> the
> > >> > head of the topic. They don't need to consume old messages they may
> > have
> > >> > missed.
> > >> >
> > >> > We would like to use the high-level consumer, as the API is a bit
> > >> simpler.
> > >> >
> > >> > Is there a way to reset the current offsets for the high-level
> > consumer?
> > >> >
> > >> > We've though about deleting the consumer group nodes in ZK manually,
> > but
> > >> > that seems brittle as those locations may change from release to
> > >> release.
> > >> >
> > >> > Is there this level of control somewhere I've missed?
> > >> >
> > >> > Thanks
> > >> > - Bob
> > >> >
> > >>
> > >
> >
> >
> > --
> > Michal Haris
> > Software Engineer
> >
> > www.visualdna.com | t: +44 (0) 207 734 7033
> >
>

It's more than that. For example, what if 2 consumers want to set different
offsets for the same partition?

Jun

On Tue, Nov 13, 2012 at 4:08 PM, Michal Haris <[EMAIL PROTECTED]>wrote:

> Of course assuming that none of the consumers in the group are interested
> in the missed messages in such case.
> On Nov 12, 2012 6:03 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
>
> > It's easy if there is only a single consumer in the group. In general,
> > however, there could be multiple consumers in the same group. So, some
> kind
> > of coordination is needed.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sat, Nov 10, 2012 at 6:41 AM, Michal Haris <
> [EMAIL PROTECTED]
> > >wrote:
> >
> > > Found it in archives, autooffset.reset is only used when the consumer
> > > registering first time or the last committed offset is out-of-range, so
> > > that means that we have to create direct connection to the zookeeper
> and
> > > set/delete the /consumer/xyz/offset/topic123/*-* before starting the
> > > consumer to get it reset to the head of the stream.. shouldn't be a big
> > > problem to patch the consumer api with a method to do that right ?
> > >
> > > On 10 November 2012 11:40, Michal Haris <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Really? So what does the autooffset.reset=largest do ?
> > > > On Nov 9, 2012 3:30 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> > > >
> > > >> Bob,
> > > >>
> > > >> The high level consumer doesn't support resetting the starting
> offset
> > > >> right
> > > >> now. We will look into that post 0.8 as part of the consumer
> redesign.
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Jun
> > > >>
> > > >>
> > > >> On Fri, Nov 9, 2012 at 5:04 AM, Bob Cotton <[EMAIL PROTECTED]>
> > > wrote:
> > > >>
> > > >> > We have a topic whose consumers, on startup, only need to consume
> > from
> > > >> the
> > > >> > head of the topic. They don't need to consume old messages they
> may
> > > have
> > > >> > missed.
> > > >> >
> > > >> > We would like to use the high-level consumer, as the API is a bit
> > > >> simpler.
> > > >> >
> > > >> > Is there a way to reset the current offsets for the high-level
> > > consumer?
> > > >> >
> > > >> > We've though about deleting the consumer group nodes in ZK
> manually,
> > > but
> > > >> > that seems brittle as those locations may change from release to
> > > >> release.
> > > >> >
> > > >> > Is there this level of control somewhere I've missed?
> > > >> >
> > > >> > Thanks
> > > >> > - Bob
> > > >> >
> > > >>
> > > >
> > >
> > >
> > > --
> > > Michal Haris
> > > Software Engineer
> > >
> > > www.visualdna.com | t: +44 (0) 207 734 7033
> > >
> >
>

Well, resetting could follow the partition ownership where only owned
partitions can be reset by any one consumer. There's definitely a case for
consumer applications that are more about observing frequencies rather than
total aggregation and so are typically interested in last couple of seconds
/ minutes of the stream.

On 14 November 2012 05:38, Jun Rao <[EMAIL PROTECTED]> wrote:

> It's more than that. For example, what if 2 consumers want to set different
> offsets for the same partition?
>
> Jun
>
> On Tue, Nov 13, 2012 at 4:08 PM, Michal Haris <[EMAIL PROTECTED]
> >wrote:
>
> > Of course assuming that none of the consumers in the group are interested
> > in the missed messages in such case.
> > On Nov 12, 2012 6:03 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> >
> > > It's easy if there is only a single consumer in the group. In general,
> > > however, there could be multiple consumers in the same group. So, some
> > kind
> > > of coordination is needed.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sat, Nov 10, 2012 at 6:41 AM, Michal Haris <
> > [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Found it in archives, autooffset.reset is only used when the consumer
> > > > registering first time or the last committed offset is out-of-range,
> so
> > > > that means that we have to create direct connection to the zookeeper
> > and
> > > > set/delete the /consumer/xyz/offset/topic123/*-* before starting the
> > > > consumer to get it reset to the head of the stream.. shouldn't be a
> big
> > > > problem to patch the consumer api with a method to do that right ?
> > > >
> > > > On 10 November 2012 11:40, Michal Haris <[EMAIL PROTECTED]>
> > > > wrote:
> > > >
> > > > > Really? So what does the autooffset.reset=largest do ?
> > > > > On Nov 9, 2012 3:30 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > >> Bob,
> > > > >>
> > > > >> The high level consumer doesn't support resetting the starting
> > offset
> > > > >> right
> > > > >> now. We will look into that post 0.8 as part of the consumer
> > redesign.
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Jun
> > > > >>
> > > > >>
> > > > >> On Fri, Nov 9, 2012 at 5:04 AM, Bob Cotton <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >>
> > > > >> > We have a topic whose consumers, on startup, only need to
> consume
> > > from
> > > > >> the
> > > > >> > head of the topic. They don't need to consume old messages they
> > may
> > > > have
> > > > >> > missed.
> > > > >> >
> > > > >> > We would like to use the high-level consumer, as the API is a
> bit
> > > > >> simpler.
> > > > >> >
> > > > >> > Is there a way to reset the current offsets for the high-level
> > > > consumer?
> > > > >> >
> > > > >> > We've though about deleting the consumer group nodes in ZK
> > manually,
> > > > but
> > > > >> > that seems brittle as those locations may change from release to
> > > > >> release.
> > > > >> >
> > > > >> > Is there this level of control somewhere I've missed?
> > > > >> >
> > > > >> > Thanks
> > > > >> > - Bob
> > > > >> >
> > > > >>
> > > > >
> > > >
> > > >
> > > > --
> > > > Michal Haris
> > > > Software Engineer
> > > >
> > > > www.visualdna.com | t: +44 (0) 207 734 7033
> > > >
> > >
> >
>

--
Michal Haris
Software Engineer

www.visualdna.com | t: +44 (0) 207 734 7033