Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> redis versus zookeeper to track consumer offsets


Copy link to this message
-
Re: redis versus zookeeper to track consumer offsets
You can do this indirectly by monitoring the avg/max latency of operations
on zookeeper. There is no direct way of measuring the requests/sec to
zookeeper since they don't expose the relevant jmx metrics.

Thanks,
Neha
On Tue, Dec 17, 2013 at 11:13 AM, S Ahmed <[EMAIL PROTECTED]> wrote:

> Interesting, wasn't aware of that.
>
> Can you comment on how you go about monitoring your ZK cluster in terms of
> throughput and if it is reaching its limits? Or is it even possible to do
> this?
>
>
> On Tue, Dec 17, 2013 at 2:01 PM, Benjamin Black <[EMAIL PROTECTED]> wrote:
>
> > ZK was designed from the start as a clustered, consistent, highly
> available
> > store for this sort of data and it works extremely well. Redis wasn't
> and I
> > don't know anyone using Redis in production, including me, who doesn't
> have
> > stories of Redis losing data. I'm sticking with ZK.
> >
> >
> > On Tue, Dec 17, 2013 at 10:57 AM, S Ahmed <[EMAIL PROTECTED]> wrote:
> >
> > > I am leaning towards using redis to track consumer offsets etc., but I
> > see
> > > how using zookeeper makes sense since it already part of the kafka
> infra.
> > >
> > > One thing which bothers me is, how are you guys keeping track of the
> load
> > > on zookeeper?  How do you get an idea when your zookeeper cluster is
> > > underprovisioned?
> > >
> > > Redis is a richer store and could help in other areas where you want to
> > > store more than just status information or offsets, and setup and
> > > administration wise it seems a bit easier to manage.
> > >
> > > Thoughts?
> > >
> >
>