Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> consumer offset reset use case


Copy link to this message
-
Re: consumer offset reset use case
You should be able to replace AsyncValue[Boolean] with an
AtomicBoolean<http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/atomic/AtomicBoolean.html>
.

As for the ZK client, maybe I don't understand your question correctly, but
I think this code is simply relying on the Zookeeper class it would find in
a standard ZK jar.

--
Felix

On Thu, May 10, 2012 at 1:00 PM, Evan Chan <[EMAIL PROTECTED]> wrote:

> Hisham,
>
> Thanks, what ZK client are you using, and is it possible to share
> AsyncValue as well?
>
> -Evan
>
>
> On Thu, May 10, 2012 at 9:43 AM, Hisham Mardam-Bey <[EMAIL PROTECTED]
> >wrote:
>
> > Mark,
> >
> > You'll find some of the code here:
> >
> > https://gist.github.com/2654382
> >
> > It uses AsyncValue which is a custom class but the rest is pretty
> > simple to modify and use for your own purposes.
> >
> > Hope this helps.
> >
> > hmb.
> >
> > On Thu, May 10, 2012 at 11:52 AM, Mark Grabois <[EMAIL PROTECTED]
> >
> > wrote:
> > > If you could that'd be great, thanks!
> > >
> > > On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]
> > >wrote:
> > >
> > >> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <
> [EMAIL PROTECTED]>
> > >> wrote:
> > >> > Hisham, thanks for the advice. Can you elaborate on how you moved
> the
> > >> > offset in ZK? I read in the archives some people cleared the offsets
> > >> > altogether. Did you access the ZK filesystem location where the
> > offsets
> > >> are
> > >> > stored:
> > >> >
> > >> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
> > >> > offset_counter_value ((persistent node)
> > >>
> > >> Mark,
> > >>
> > >> I've tried both approaches:
> > >>
> > >> - deleted the ZK nodes that hold the offsets
> > >> - reset the value of those nodes to zero
> > >>
> > >> Both worked however I stuck with deleting the nodes (since the files
> > >> represent broker/partition). I can share some of the code if you'd
> > >> like.
> > >>
> > >> hmb.
> > >>
> > >> >
> > >> >
> > >> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <
> > [EMAIL PROTECTED]
> > >> >wrote:
> > >> >
> > >> >> Mark,
> > >> >>
> > >> >> We have similar needs and have added calls to our code to stop
> > >> >> consumption, move the offset in ZK, then resume (we use it to go
> back
> > >> >> in time).
> > >> >>
> > >> >> hmb.
> > >> >>
> > >> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <
> > [EMAIL PROTECTED]
> > >> >
> > >> >> wrote:
> > >> >> > Hello,
> > >> >> >
> > >> >> > I know the topic of consumer offsets has come up frequently on
> the
> > >> list,
> > >> >> > how it is not possible to control the offset without using
> > >> >> SimpleConsumer.
> > >> >> > I would like to be able to reset all consumer offsets to
> "largest"
> > (so
> > >> >> that
> > >> >> > consumption may start with the latest produced messages) when
> > >> consumption
> > >> >> > begins to lag behind our data feed. Is there a good way to do it
> > with
> > >> >> > zookeeper (since we will be using multiple consumers), without
> > >> restarting
> > >> >> > consumer?
> > >> >> >
> > >> >> > Thank you,
> > >> >> > Mark
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Hisham Mardam-Bey
> > >> >> [ Director of Engineering ] [ Mate1 Inc. ]
> > >> >>
> > >> >> A: Because it messes up the order in which people normally read
> text.
> > >> >> Q: Why is top-posting such a bad thing?
> > >> >> A: Top-posting.
> > >> >> Q: What is the most annoying thing in e-mail?
> > >> >>
> > >> >> -=[ Codito Ergo Sum ]=-
> > >> >>
> > >>
> > >>
> > >>
> > >> --
> > >> Hisham Mardam-Bey
> > >> [ Director of Engineering ] [ Mate1 Inc. ]
> > >>
> > >> A: Because it messes up the order in which people normally read text.
> > >> Q: Why is top-posting such a bad thing?
> > >> A: Top-posting.
> > >> Q: What is the most annoying thing in e-mail?
> > >>
> > >> -=[ Codito Ergo Sum ]=-
> > >>
> >
> >
> >
> > --
> > Hisham Mardam-Bey
> > [ Director of Engineering ] [ Mate1 Inc. ]
> >
> > A: Because it messes up the order in which people normally read text.