Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> consumer offset reset use case


Copy link to this message
-
Re: consumer offset reset use case
You should be able to replace AsyncValue[Boolean] with an
AtomicBoolean<http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/atomic/AtomicBoolean.html>
.

As for the ZK client, maybe I don't understand your question correctly, but
I think this code is simply relying on the Zookeeper class it would find in
a standard ZK jar.

--
Felix

On Thu, May 10, 2012 at 1:00 PM, Evan Chan <[EMAIL PROTECTED]> wrote:

> Hisham,
>
> Thanks, what ZK client are you using, and is it possible to share
> AsyncValue as well?
>
> -Evan
>
>
> On Thu, May 10, 2012 at 9:43 AM, Hisham Mardam-Bey <[EMAIL PROTECTED]
> >wrote:
>
> > Mark,
> >
> > You'll find some of the code here:
> >
> > https://gist.github.com/2654382
> >
> > It uses AsyncValue which is a custom class but the rest is pretty
> > simple to modify and use for your own purposes.
> >
> > Hope this helps.
> >
> > hmb.
> >
> > On Thu, May 10, 2012 at 11:52 AM, Mark Grabois <[EMAIL PROTECTED]
> >
> > wrote:
> > > If you could that'd be great, thanks!
> > >
> > > On Wed, May 9, 2012 at 8:26 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]
> > >wrote:
> > >
> > >> On Wed, May 9, 2012 at 3:18 PM, Mark Grabois <
> [EMAIL PROTECTED]>
> > >> wrote:
> > >> > Hisham, thanks for the advice. Can you elaborate on how you moved
> the
> > >> > offset in ZK? I read in the archives some people cleared the offsets
> > >> > altogether. Did you access the ZK filesystem location where the
> > offsets
> > >> are
> > >> > stored:
> > >> >
> > >> > /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] -->
> > >> > offset_counter_value ((persistent node)
> > >>
> > >> Mark,
> > >>
> > >> I've tried both approaches:
> > >>
> > >> - deleted the ZK nodes that hold the offsets
> > >> - reset the value of those nodes to zero
> > >>
> > >> Both worked however I stuck with deleting the nodes (since the files
> > >> represent broker/partition). I can share some of the code if you'd
> > >> like.
> > >>
> > >> hmb.
> > >>
> > >> >
> > >> >
> > >> > On Tue, May 8, 2012 at 11:20 PM, Hisham Mardam-Bey <
> > [EMAIL PROTECTED]
> > >> >wrote:
> > >> >
> > >> >> Mark,
> > >> >>
> > >> >> We have similar needs and have added calls to our code to stop
> > >> >> consumption, move the offset in ZK, then resume (we use it to go
> back
> > >> >> in time).
> > >> >>
> > >> >> hmb.
> > >> >>
> > >> >> On Sun, May 6, 2012 at 11:36 PM, Mark Grabois <
> > [EMAIL PROTECTED]
> > >> >
> > >> >> wrote:
> > >> >> > Hello,
> > >> >> >
> > >> >> > I know the topic of consumer offsets has come up frequently on
> the
> > >> list,
> > >> >> > how it is not possible to control the offset without using
> > >> >> SimpleConsumer.
> > >> >> > I would like to be able to reset all consumer offsets to
> "largest"
> > (so
> > >> >> that
> > >> >> > consumption may start with the latest produced messages) when
> > >> consumption
> > >> >> > begins to lag behind our data feed. Is there a good way to do it
> > with
> > >> >> > zookeeper (since we will be using multiple consumers), without
> > >> restarting
> > >> >> > consumer?
> > >> >> >
> > >> >> > Thank you,
> > >> >> > Mark
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Hisham Mardam-Bey
> > >> >> [ Director of Engineering ] [ Mate1 Inc. ]
> > >> >>
> > >> >> A: Because it messes up the order in which people normally read
> text.
> > >> >> Q: Why is top-posting such a bad thing?
> > >> >> A: Top-posting.
> > >> >> Q: What is the most annoying thing in e-mail?
> > >> >>
> > >> >> -=[ Codito Ergo Sum ]=-
> > >> >>
> > >>
> > >>
> > >>
> > >> --
> > >> Hisham Mardam-Bey
> > >> [ Director of Engineering ] [ Mate1 Inc. ]
> > >>
> > >> A: Because it messes up the order in which people normally read text.
> > >> Q: Why is top-posting such a bad thing?
> > >> A: Top-posting.
> > >> Q: What is the most annoying thing in e-mail?
> > >>
> > >> -=[ Codito Ergo Sum ]=-
> > >>
> >
> >
> >
> > --
> > Hisham Mardam-Bey
> > [ Director of Engineering ] [ Mate1 Inc. ]
> >
> > A: Because it messes up the order in which people normally read text.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB