Hello,

I know the topic of consumer offsets has come up frequently on the list,
how it is not possible to control the offset without using SimpleConsumer.
I would like to be able to reset all consumer offsets to "largest" (so that
consumption may start with the latest produced messages) when consumption
begins to lag behind our data feed. Is there a good way to do it with
zookeeper (since we will be using multiple consumers), without restarting
consumer?

Thank you,
Mark

Mark,

Currently, there is no reset API in the consumer. We are thinking about how
to add that properly in the future. For your use case, you can potentially
increase the degree of parallelism in the consumer to drive up the
consumption rate or throttle the consumption in your app.

Thanks,

Jun

On Sun, May 6, 2012 at 8:36 PM, Mark Grabois <[EMAIL PROTECTED]>wrote:

> Hello,
>
> I know the topic of consumer offsets has come up frequently on the list,
> how it is not possible to control the offset without using SimpleConsumer.
> I would like to be able to reset all consumer offsets to "largest" (so that
> consumption may start with the latest produced messages) when consumption
> begins to lag behind our data feed. Is there a good way to do it with
> zookeeper (since we will be using multiple consumers), without restarting
> consumer?
>
> Thank you,
> Mark
>

Mark,

We have similar needs and have added calls to our code to stop
consumption, move the offset in ZK, then resume (we use it to go back
in time).

hmb.

On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <[EMAIL PROTECTED]> wrote:
> Hello,
>
> I know the topic of consumer offsets has come up frequently on the list,
> how it is not possible to control the offset without using SimpleConsumer.
> I would like to be able to reset all consumer offsets to "largest" (so that
> consumption may start with the latest produced messages) when consumption
> begins to lag behind our data feed. Is there a good way to do it with
> zookeeper (since we will be using multiple consumers), without restarting
> consumer?
>
> Thank you,
> Mark

--
Hisham Mardam-Bey
[ Director of Engineering ] [ Mate1 Inc. ]

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

-=[ Codito Ergo Sum ]=-

Hisham, thanks for the advice. Can you elaborate on how you moved the
offset in ZK? I read in the archives some people cleared the offsets
altogether. Did you access the ZK filesystem location where the offsets are
stored:

/consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
offset_counter_value ((persistent node)

On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]>wrote:

> Mark,
>
> We have similar needs and have added calls to our code to stop
> consumption, move the offset in ZK, then resume (we use it to go back
> in time).
>
> hmb.
>
> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <[EMAIL PROTECTED]>
> wrote:
> > Hello,
> >
> > I know the topic of consumer offsets has come up frequently on the list,
> > how it is not possible to control the offset without using
> SimpleConsumer.
> > I would like to be able to reset all consumer offsets to "largest" (so
> that
> > consumption may start with the latest produced messages) when consumption
> > begins to lag behind our data feed. Is there a good way to do it with
> > zookeeper (since we will be using multiple consumers), without restarting
> > consumer?
> >
> > Thank you,
> > Mark
>
>
>
> --
> Hisham Mardam-Bey
> [ Director of Engineering ] [ Mate1 Inc. ]
>
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?
>
> -=[ Codito Ergo Sum ]=-
>

On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <[EMAIL PROTECTED]> wrote:
> Hisham, thanks for the advice. Can you elaborate on how you moved the
> offset in ZK? I read in the archives some people cleared the offsets
> altogether. Did you access the ZK filesystem location where the offsets are
> stored:
>
> /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
> offset_counter_value ((persistent node)

Mark,

I've tried both approaches:

- deleted the ZK nodes that hold the offsets
- reset the value of those nodes to zero

Both worked however I stuck with deleting the nodes (since the files
represent broker/partition). I can share some of the code if you'd
like.

hmb.

>
>
> On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]>wrote:
>
>> Mark,
>>
>> We have similar needs and have added calls to our code to stop
>> consumption, move the offset in ZK, then resume (we use it to go back
>> in time).
>>
>> hmb.
>>
>> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <[EMAIL PROTECTED]>
>> wrote:
>> > Hello,
>> >
>> > I know the topic of consumer offsets has come up frequently on the list,
>> > how it is not possible to control the offset without using
>> SimpleConsumer.
>> > I would like to be able to reset all consumer offsets to "largest" (so
>> that
>> > consumption may start with the latest produced messages) when consumption
>> > begins to lag behind our data feed. Is there a good way to do it with
>> > zookeeper (since we will be using multiple consumers), without restarting
>> > consumer?
>> >
>> > Thank you,
>> > Mark
>>
>>
>>
>> --
>> Hisham Mardam-Bey
>> [ Director of Engineering ] [ Mate1 Inc. ]
>>
>> A: Because it messes up the order in which people normally read text.
>> Q: Why is top-posting such a bad thing?
>> A: Top-posting.
>> Q: What is the most annoying thing in e-mail?
>>
>> -=[ Codito Ergo Sum ]=-
>>

--
Hisham Mardam-Bey
[ Director of Engineering ] [ Mate1 Inc. ]

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

-=[ Codito Ergo Sum ]=-

If you could that'd be great, thanks!

On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]>wrote:

> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <[EMAIL PROTECTED]>
> wrote:
> > Hisham, thanks for the advice. Can you elaborate on how you moved the
> > offset in ZK? I read in the archives some people cleared the offsets
> > altogether. Did you access the ZK filesystem location where the offsets
> are
> > stored:
> >
> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
> > offset_counter_value ((persistent node)
>
> Mark,
>
> I've tried both approaches:
>
> - deleted the ZK nodes that hold the offsets
> - reset the value of those nodes to zero
>
> Both worked however I stuck with deleting the nodes (since the files
> represent broker/partition). I can share some of the code if you'd
> like.
>
> hmb.
>
> >
> >
> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]
> >wrote:
> >
> >> Mark,
> >>
> >> We have similar needs and have added calls to our code to stop
> >> consumption, move the offset in ZK, then resume (we use it to go back
> >> in time).
> >>
> >> hmb.
> >>
> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <[EMAIL PROTECTED]
> >
> >> wrote:
> >> > Hello,
> >> >
> >> > I know the topic of consumer offsets has come up frequently on the
> list,
> >> > how it is not possible to control the offset without using
> >> SimpleConsumer.
> >> > I would like to be able to reset all consumer offsets to "largest" (so
> >> that
> >> > consumption may start with the latest produced messages) when
> consumption
> >> > begins to lag behind our data feed. Is there a good way to do it with
> >> > zookeeper (since we will be using multiple consumers), without
> restarting
> >> > consumer?
> >> >
> >> > Thank you,
> >> > Mark
> >>
> >>
> >>
> >> --
> >> Hisham Mardam-Bey
> >> [ Director of Engineering ] [ Mate1 Inc. ]
> >>
> >> A: Because it messes up the order in which people normally read text.
> >> Q: Why is top-posting such a bad thing?
> >> A: Top-posting.
> >> Q: What is the most annoying thing in e-mail?
> >>
> >> -=[ Codito Ergo Sum ]=-
> >>
>
>
>
> --
> Hisham Mardam-Bey
> [ Director of Engineering ] [ Mate1 Inc. ]
>
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?
>
> -=[ Codito Ergo Sum ]=-
>

Mark,

You'll find some of the code here:

https://gist.github.com/2654382

It uses AsyncValue which is a custom class but the rest is pretty
simple to modify and use for your own purposes.

Hope this helps.

hmb.

On Thu, May 10, 2012 at 11:52 AM, Mark Grabois <[EMAIL PROTECTED]> wrote:
> If you could that'd be great, thanks!
>
> On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]>wrote:
>
>> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <[EMAIL PROTECTED]>
>> wrote:
>> > Hisham, thanks for the advice. Can you elaborate on how you moved the
>> > offset in ZK? I read in the archives some people cleared the offsets
>> > altogether. Did you access the ZK filesystem location where the offsets
>> are
>> > stored:
>> >
>> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
>> > offset_counter_value ((persistent node)
>>
>> Mark,
>>
>> I've tried both approaches:
>>
>> - deleted the ZK nodes that hold the offsets
>> - reset the value of those nodes to zero
>>
>> Both worked however I stuck with deleting the nodes (since the files
>> represent broker/partition). I can share some of the code if you'd
>> like.
>>
>> hmb.
>>
>> >
>> >
>> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]
>> >wrote:
>> >
>> >> Mark,
>> >>
>> >> We have similar needs and have added calls to our code to stop
>> >> consumption, move the offset in ZK, then resume (we use it to go back
>> >> in time).
>> >>
>> >> hmb.
>> >>
>> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <[EMAIL PROTECTED]
>> >
>> >> wrote:
>> >> > Hello,
>> >> >
>> >> > I know the topic of consumer offsets has come up frequently on the
>> list,
>> >> > how it is not possible to control the offset without using
>> >> SimpleConsumer.
>> >> > I would like to be able to reset all consumer offsets to "largest" (so
>> >> that
>> >> > consumption may start with the latest produced messages) when
>> consumption
>> >> > begins to lag behind our data feed. Is there a good way to do it with
>> >> > zookeeper (since we will be using multiple consumers), without
>> restarting
>> >> > consumer?
>> >> >
>> >> > Thank you,
>> >> > Mark
>> >>
>> >>
>> >>
>> >> --
>> >> Hisham Mardam-Bey
>> >> [ Director of Engineering ] [ Mate1 Inc. ]
>> >>
>> >> A: Because it messes up the order in which people normally read text.
>> >> Q: Why is top-posting such a bad thing?
>> >> A: Top-posting.
>> >> Q: What is the most annoying thing in e-mail?
>> >>
>> >> -=[ Codito Ergo Sum ]=-
>> >>
>>
>>
>>
>> --
>> Hisham Mardam-Bey
>> [ Director of Engineering ] [ Mate1 Inc. ]
>>
>> A: Because it messes up the order in which people normally read text.
>> Q: Why is top-posting such a bad thing?
>> A: Top-posting.
>> Q: What is the most annoying thing in e-mail?
>>
>> -=[ Codito Ergo Sum ]=-
>>

--
Hisham Mardam-Bey
[ Director of Engineering ] [ Mate1 Inc. ]

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

-=[ Codito Ergo Sum ]=-

Hisham,

Thanks, what ZK client are you using, and is it possible to share
AsyncValue as well?

-Evan
On Thu, May 10, 2012 at 9:43 AM, Hisham Mardam-Bey <[EMAIL PROTECTED]>wrote:

> Mark,
>
> You'll find some of the code here:
>
> https://gist.github.com/2654382
>
> It uses AsyncValue which is a custom class but the rest is pretty
> simple to modify and use for your own purposes.
>
> Hope this helps.
>
> hmb.
>
> On Thu, May 10, 2012 at 11:52 AM, Mark Grabois <[EMAIL PROTECTED]>
> wrote:
> > If you could that'd be great, thanks!
> >
> > On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]
> >wrote:
> >
> >> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <[EMAIL PROTECTED]>
> >> wrote:
> >> > Hisham, thanks for the advice. Can you elaborate on how you moved the
> >> > offset in ZK? I read in the archives some people cleared the offsets
> >> > altogether. Did you access the ZK filesystem location where the
> offsets
> >> are
> >> > stored:
> >> >
> >> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
> >> > offset_counter_value ((persistent node)
> >>
> >> Mark,
> >>
> >> I've tried both approaches:
> >>
> >> - deleted the ZK nodes that hold the offsets
> >> - reset the value of those nodes to zero
> >>
> >> Both worked however I stuck with deleting the nodes (since the files
> >> represent broker/partition). I can share some of the code if you'd
> >> like.
> >>
> >> hmb.
> >>
> >> >
> >> >
> >> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <
> [EMAIL PROTECTED]
> >> >wrote:
> >> >
> >> >> Mark,
> >> >>
> >> >> We have similar needs and have added calls to our code to stop
> >> >> consumption, move the offset in ZK, then resume (we use it to go back
> >> >> in time).
> >> >>
> >> >> hmb.
> >> >>
> >> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <
> [EMAIL PROTECTED]
> >> >
> >> >> wrote:
> >> >> > Hello,
> >> >> >
> >> >> > I know the topic of consumer offsets has come up frequently on the
> >> list,
> >> >> > how it is not possible to control the offset without using
> >> >> SimpleConsumer.
> >> >> > I would like to be able to reset all consumer offsets to "largest"
> (so
> >> >> that
> >> >> > consumption may start with the latest produced messages) when
> >> consumption
> >> >> > begins to lag behind our data feed. Is there a good way to do it
> with
> >> >> > zookeeper (since we will be using multiple consumers), without
> >> restarting
> >> >> > consumer?
> >> >> >
> >> >> > Thank you,
> >> >> > Mark
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Hisham Mardam-Bey
> >> >> [ Director of Engineering ] [ Mate1 Inc. ]
> >> >>
> >> >> A: Because it messes up the order in which people normally read text.
> >> >> Q: Why is top-posting such a bad thing?
> >> >> A: Top-posting.
> >> >> Q: What is the most annoying thing in e-mail?
> >> >>
> >> >> -=[ Codito Ergo Sum ]=-
> >> >>
> >>
> >>
> >>
> >> --
> >> Hisham Mardam-Bey
> >> [ Director of Engineering ] [ Mate1 Inc. ]
> >>
> >> A: Because it messes up the order in which people normally read text.
> >> Q: Why is top-posting such a bad thing?
> >> A: Top-posting.
> >> Q: What is the most annoying thing in e-mail?
> >>
> >> -=[ Codito Ergo Sum ]=-
> >>
>
>
>
> --
> Hisham Mardam-Bey
> [ Director of Engineering ] [ Mate1 Inc. ]
>
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?
>
> -=[ Codito Ergo Sum ]=-
>

--
--
*Evan Chan*
Senior Software Engineer |
[EMAIL PROTECTED] | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>

You should be able to replace AsyncValue[Boolean] with an
AtomicBoolean<http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/atomic/AtomicBoolean.html>
.

As for the ZK client, maybe I don't understand your question correctly, but
I think this code is simply relying on the Zookeeper class it would find in
a standard ZK jar.

--
Felix

On Thu, May 10, 2012 at 1:00 PM, Evan Chan <[EMAIL PROTECTED]> wrote:

> Hisham,
>
> Thanks, what ZK client are you using, and is it possible to share
> AsyncValue as well?
>
> -Evan
>
>
> On Thu, May 10, 2012 at 9:43 AM, Hisham Mardam-Bey <[EMAIL PROTECTED]
> >wrote:
>
> > Mark,
> >
> > You'll find some of the code here:
> >
> > https://gist.github.com/2654382
> >
> > It uses AsyncValue which is a custom class but the rest is pretty
> > simple to modify and use for your own purposes.
> >
> > Hope this helps.
> >
> > hmb.
> >
> > On Thu, May 10, 2012 at 11:52 AM, Mark Grabois <[EMAIL PROTECTED]
> >
> > wrote:
> > > If you could that'd be great, thanks!
> > >
> > > On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]
> > >wrote:
> > >
> > >> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <
> [EMAIL PROTECTED]>
> > >> wrote:
> > >> > Hisham, thanks for the advice. Can you elaborate on how you moved
> the
> > >> > offset in ZK? I read in the archives some people cleared the offsets
> > >> > altogether. Did you access the ZK filesystem location where the
> > offsets
> > >> are
> > >> > stored:
> > >> >
> > >> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
> > >> > offset_counter_value ((persistent node)
> > >>
> > >> Mark,
> > >>
> > >> I've tried both approaches:
> > >>
> > >> - deleted the ZK nodes that hold the offsets
> > >> - reset the value of those nodes to zero
> > >>
> > >> Both worked however I stuck with deleting the nodes (since the files
> > >> represent broker/partition). I can share some of the code if you'd
> > >> like.
> > >>
> > >> hmb.
> > >>
> > >> >
> > >> >
> > >> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <
> > [EMAIL PROTECTED]
> > >> >wrote:
> > >> >
> > >> >> Mark,
> > >> >>
> > >> >> We have similar needs and have added calls to our code to stop
> > >> >> consumption, move the offset in ZK, then resume (we use it to go
> back
> > >> >> in time).
> > >> >>
> > >> >> hmb.
> > >> >>
> > >> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <
> > [EMAIL PROTECTED]
> > >> >
> > >> >> wrote:
> > >> >> > Hello,
> > >> >> >
> > >> >> > I know the topic of consumer offsets has come up frequently on
> the
> > >> list,
> > >> >> > how it is not possible to control the offset without using
> > >> >> SimpleConsumer.
> > >> >> > I would like to be able to reset all consumer offsets to
> "largest"
> > (so
> > >> >> that
> > >> >> > consumption may start with the latest produced messages) when
> > >> consumption
> > >> >> > begins to lag behind our data feed. Is there a good way to do it
> > with
> > >> >> > zookeeper (since we will be using multiple consumers), without
> > >> restarting
> > >> >> > consumer?
> > >> >> >
> > >> >> > Thank you,
> > >> >> > Mark
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Hisham Mardam-Bey
> > >> >> [ Director of Engineering ] [ Mate1 Inc. ]
> > >> >>
> > >> >> A: Because it messes up the order in which people normally read
> text.
> > >> >> Q: Why is top-posting such a bad thing?
> > >> >> A: Top-posting.
> > >> >> Q: What is the most annoying thing in e-mail?
> > >> >>
> > >> >> -=[ Codito Ergo Sum ]=-
> > >> >>
> > >>
> > >>
> > >>
> > >> --
> > >> Hisham Mardam-Bey
> > >> [ Director of Engineering ] [ Mate1 Inc. ]
> > >>
> > >> A: Because it messes up the order in which people normally read text.
> > >> Q: Why is top-posting such a bad thing?
> > >> A: Top-posting.
> > >> Q: What is the most annoying thing in e-mail?
> > >>
> > >> -=[ Codito Ergo Sum ]=-
> > >>
> >
> >
> >
> > --
> > Hisham Mardam-Bey
> > [ Director of Engineering ] [ Mate1 Inc. ]
> >
> > A: Because it messes up the order in which people normally read text.

Evan,

AsyncValue is here:

https://gist.github.com/2663670

I'm using the ZK API from org.apache.zookeeper.ZooKeeper.

hmb.

On Thu, May 10, 2012 at 1:00 PM, Evan Chan <[EMAIL PROTECTED]> wrote:
> Hisham,
>
> Thanks, what ZK client are you using, and is it possible to share
> AsyncValue as well?
>
> -Evan
>
>
> On Thu, May 10, 2012 at 9:43 AM, Hisham Mardam-Bey <[EMAIL PROTECTED]>wrote:
>
>> Mark,
>>
>> You'll find some of the code here:
>>
>> https://gist.github.com/2654382
>>
>> It uses AsyncValue which is a custom class but the rest is pretty
>> simple to modify and use for your own purposes.
>>
>> Hope this helps.
>>
>> hmb.
>>
>> On Thu, May 10, 2012 at 11:52 AM, Mark Grabois <[EMAIL PROTECTED]>
>> wrote:
>> > If you could that'd be great, thanks!
>> >
>> > On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]
>> >wrote:
>> >
>> >> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > Hisham, thanks for the advice. Can you elaborate on how you moved the
>> >> > offset in ZK? I read in the archives some people cleared the offsets
>> >> > altogether. Did you access the ZK filesystem location where the
>> offsets
>> >> are
>> >> > stored:
>> >> >
>> >> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
>> >> > offset_counter_value ((persistent node)
>> >>
>> >> Mark,
>> >>
>> >> I've tried both approaches:
>> >>
>> >> - deleted the ZK nodes that hold the offsets
>> >> - reset the value of those nodes to zero
>> >>
>> >> Both worked however I stuck with deleting the nodes (since the files
>> >> represent broker/partition). I can share some of the code if you'd
>> >> like.
>> >>
>> >> hmb.
>> >>
>> >> >
>> >> >
>> >> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <
>> [EMAIL PROTECTED]
>> >> >wrote:
>> >> >
>> >> >> Mark,
>> >> >>
>> >> >> We have similar needs and have added calls to our code to stop
>> >> >> consumption, move the offset in ZK, then resume (we use it to go back
>> >> >> in time).
>> >> >>
>> >> >> hmb.
>> >> >>
>> >> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <
>> [EMAIL PROTECTED]
>> >> >
>> >> >> wrote:
>> >> >> > Hello,
>> >> >> >
>> >> >> > I know the topic of consumer offsets has come up frequently on the
>> >> list,
>> >> >> > how it is not possible to control the offset without using
>> >> >> SimpleConsumer.
>> >> >> > I would like to be able to reset all consumer offsets to "largest"
>> (so
>> >> >> that
>> >> >> > consumption may start with the latest produced messages) when
>> >> consumption
>> >> >> > begins to lag behind our data feed. Is there a good way to do it
>> with
>> >> >> > zookeeper (since we will be using multiple consumers), without
>> >> restarting
>> >> >> > consumer?
>> >> >> >
>> >> >> > Thank you,
>> >> >> > Mark
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Hisham Mardam-Bey
>> >> >> [ Director of Engineering ] [ Mate1 Inc. ]
>> >> >>
>> >> >> A: Because it messes up the order in which people normally read text.
>> >> >> Q: Why is top-posting such a bad thing?
>> >> >> A: Top-posting.
>> >> >> Q: What is the most annoying thing in e-mail?
>> >> >>
>> >> >> -=[ Codito Ergo Sum ]=-
>> >> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Hisham Mardam-Bey
>> >> [ Director of Engineering ] [ Mate1 Inc. ]
>> >>
>> >> A: Because it messes up the order in which people normally read text.
>> >> Q: Why is top-posting such a bad thing?
>> >> A: Top-posting.
>> >> Q: What is the most annoying thing in e-mail?
>> >>
>> >> -=[ Codito Ergo Sum ]=-
>> >>
>>
>>
>>
>> --
>> Hisham Mardam-Bey
>> [ Director of Engineering ] [ Mate1 Inc. ]
>>
>> A: Because it messes up the order in which people normally read text.
>> Q: Why is top-posting such a bad thing?
>> A: Top-posting.
>> Q: What is the most annoying thing in e-mail?
>>
>> -=[ Codito Ergo Sum ]=-
>>
>
>
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> [EMAIL PROTECTED] | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>

Hisham Mardam-Bey
[ Director of Engineering ] [ Mate1 Inc. ]

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

-=[ Codito Ergo Sum ]=-

Hisham,

Thanks.
A similar concept is Akka Agents  (ever since discovering Akka, I haven't
gone back to using the built in Actors library):
http://doc.akka.io/docs/akka/2.0.1/scala/agents.html

-Evan
On Fri, May 11, 2012 at 6:54 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]>wrote:

> Evan,
>
> AsyncValue is here:
>
> https://gist.github.com/2663670
>
> I'm using the ZK API from org.apache.zookeeper.ZooKeeper.
>
> hmb.
>
> On Thu, May 10, 2012 at 1:00 PM, Evan Chan <[EMAIL PROTECTED]> wrote:
> > Hisham,
> >
> > Thanks, what ZK client are you using, and is it possible to share
> > AsyncValue as well?
> >
> > -Evan
> >
> >
> > On Thu, May 10, 2012 at 9:43 AM, Hisham Mardam-Bey <[EMAIL PROTECTED]
> >wrote:
> >
> >> Mark,
> >>
> >> You'll find some of the code here:
> >>
> >> https://gist.github.com/2654382
> >>
> >> It uses AsyncValue which is a custom class but the rest is pretty
> >> simple to modify and use for your own purposes.
> >>
> >> Hope this helps.
> >>
> >> hmb.
> >>
> >> On Thu, May 10, 2012 at 11:52 AM, Mark Grabois <
> [EMAIL PROTECTED]>
> >> wrote:
> >> > If you could that'd be great, thanks!
> >> >
> >> > On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <
> [EMAIL PROTECTED]
> >> >wrote:
> >> >
> >> >> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <
> [EMAIL PROTECTED]>
> >> >> wrote:
> >> >> > Hisham, thanks for the advice. Can you elaborate on how you moved
> the
> >> >> > offset in ZK? I read in the archives some people cleared the
> offsets
> >> >> > altogether. Did you access the ZK filesystem location where the
> >> offsets
> >> >> are
> >> >> > stored:
> >> >> >
> >> >> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
> >> >> > offset_counter_value ((persistent node)
> >> >>
> >> >> Mark,
> >> >>
> >> >> I've tried both approaches:
> >> >>
> >> >> - deleted the ZK nodes that hold the offsets
> >> >> - reset the value of those nodes to zero
> >> >>
> >> >> Both worked however I stuck with deleting the nodes (since the files
> >> >> represent broker/partition). I can share some of the code if you'd
> >> >> like.
> >> >>
> >> >> hmb.
> >> >>
> >> >> >
> >> >> >
> >> >> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <
> >> [EMAIL PROTECTED]
> >> >> >wrote:
> >> >> >
> >> >> >> Mark,
> >> >> >>
> >> >> >> We have similar needs and have added calls to our code to stop
> >> >> >> consumption, move the offset in ZK, then resume (we use it to go
> back
> >> >> >> in time).
> >> >> >>
> >> >> >> hmb.
> >> >> >>
> >> >> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <
> >> [EMAIL PROTECTED]
> >> >> >
> >> >> >> wrote:
> >> >> >> > Hello,
> >> >> >> >
> >> >> >> > I know the topic of consumer offsets has come up frequently on
> the
> >> >> list,
> >> >> >> > how it is not possible to control the offset without using
> >> >> >> SimpleConsumer.
> >> >> >> > I would like to be able to reset all consumer offsets to
> "largest"
> >> (so
> >> >> >> that
> >> >> >> > consumption may start with the latest produced messages) when
> >> >> consumption
> >> >> >> > begins to lag behind our data feed. Is there a good way to do it
> >> with
> >> >> >> > zookeeper (since we will be using multiple consumers), without
> >> >> restarting
> >> >> >> > consumer?
> >> >> >> >
> >> >> >> > Thank you,
> >> >> >> > Mark
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Hisham Mardam-Bey
> >> >> >> [ Director of Engineering ] [ Mate1 Inc. ]
> >> >> >>
> >> >> >> A: Because it messes up the order in which people normally read
> text.
> >> >> >> Q: Why is top-posting such a bad thing?
> >> >> >> A: Top-posting.
> >> >> >> Q: What is the most annoying thing in e-mail?
> >> >> >>
> >> >> >> -=[ Codito Ergo Sum ]=-
> >> >> >>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Hisham Mardam-Bey
> >> >> [ Director of Engineering ] [ Mate1 Inc. ]
> >> >>
> >> >> A: Because it messes up the order in which people normally read text.
> >> >> Q: Why is top-posting such a bad thing?
> >> >> A: Top-posting.

*Evan Chan*
Senior Software Engineer |
[EMAIL PROTECTED] | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>

Hi Evan,

Exactly! We're deprecating our regular Scala actors for Akka's now
that 2.x is out in the wild (
hmb.

On Mon, May 14, 2012 at 4:02 AM, Evan Chan <[EMAIL PROTECTED]> wrote:
> Hisham,
>
> Thanks.
> A similar concept is Akka Agents  (ever since discovering Akka, I haven't
> gone back to using the built in Actors library):
> http://doc.akka.io/docs/akka/2.0.1/scala/agents.html
>
> -Evan
>
>
> On Fri, May 11, 2012 at 6:54 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]>wrote:
>
>> Evan,
>>
>> AsyncValue is here:
>>
>> https://gist.github.com/2663670
>>
>> I'm using the ZK API from org.apache.zookeeper.ZooKeeper.
>>
>> hmb.
>>
>> On Thu, May 10, 2012 at 1:00 PM, Evan Chan <[EMAIL PROTECTED]> wrote:
>> > Hisham,
>> >
>> > Thanks, what ZK client are you using, and is it possible to share
>> > AsyncValue as well?
>> >
>> > -Evan
>> >
>> >
>> > On Thu, May 10, 2012 at 9:43 AM, Hisham Mardam-Bey <[EMAIL PROTECTED]
>> >wrote:
>> >
>> >> Mark,
>> >>
>> >> You'll find some of the code here:
>> >>
>> >> https://gist.github.com/2654382
>> >>
>> >> It uses AsyncValue which is a custom class but the rest is pretty
>> >> simple to modify and use for your own purposes.
>> >>
>> >> Hope this helps.
>> >>
>> >> hmb.
>> >>
>> >> On Thu, May 10, 2012 at 11:52 AM, Mark Grabois <
>> [EMAIL PROTECTED]>
>> >> wrote:
>> >> > If you could that'd be great, thanks!
>> >> >
>> >> > On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <
>> [EMAIL PROTECTED]
>> >> >wrote:
>> >> >
>> >> >> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <
>> [EMAIL PROTECTED]>
>> >> >> wrote:
>> >> >> > Hisham, thanks for the advice. Can you elaborate on how you moved
>> the
>> >> >> > offset in ZK? I read in the archives some people cleared the
>> offsets
>> >> >> > altogether. Did you access the ZK filesystem location where the
>> >> offsets
>> >> >> are
>> >> >> > stored:
>> >> >> >
>> >> >> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
>> >> >> > offset_counter_value ((persistent node)
>> >> >>
>> >> >> Mark,
>> >> >>
>> >> >> I've tried both approaches:
>> >> >>
>> >> >> - deleted the ZK nodes that hold the offsets
>> >> >> - reset the value of those nodes to zero
>> >> >>
>> >> >> Both worked however I stuck with deleting the nodes (since the files
>> >> >> represent broker/partition). I can share some of the code if you'd
>> >> >> like.
>> >> >>
>> >> >> hmb.
>> >> >>
>> >> >> >
>> >> >> >
>> >> >> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <
>> >> [EMAIL PROTECTED]
>> >> >> >wrote:
>> >> >> >
>> >> >> >> Mark,
>> >> >> >>
>> >> >> >> We have similar needs and have added calls to our code to stop
>> >> >> >> consumption, move the offset in ZK, then resume (we use it to go
>> back
>> >> >> >> in time).
>> >> >> >>
>> >> >> >> hmb.
>> >> >> >>
>> >> >> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <
>> >> [EMAIL PROTECTED]
>> >> >> >
>> >> >> >> wrote:
>> >> >> >> > Hello,
>> >> >> >> >
>> >> >> >> > I know the topic of consumer offsets has come up frequently on
>> the
>> >> >> list,
>> >> >> >> > how it is not possible to control the offset without using
>> >> >> >> SimpleConsumer.
>> >> >> >> > I would like to be able to reset all consumer offsets to
>> "largest"
>> >> (so
>> >> >> >> that
>> >> >> >> > consumption may start with the latest produced messages) when
>> >> >> consumption
>> >> >> >> > begins to lag behind our data feed. Is there a good way to do it
>> >> with
>> >> >> >> > zookeeper (since we will be using multiple consumers), without
>> >> >> restarting
>> >> >> >> > consumer?
>> >> >> >> >
>> >> >> >> > Thank you,
>> >> >> >> > Mark
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> Hisham Mardam-Bey
>> >> >> >> [ Director of Engineering ] [ Mate1 Inc. ]
>> >> >> >>
>> >> >> >> A: Because it messes up the order in which people normally read
>> text.
>> >> >> >> Q: Why is top-posting such a bad thing?
>> >> >> >> A: Top-posting.
>> >> >> >> Q: What is the most annoying thing in e-mail?
>> >> >> >>

Hisham Mardam-Bey
[ Director of Engineering ] [ Mate1 Inc. ]

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

-=[ Codito Ergo Sum ]=-