Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> Multi threaded queue consumers


Copy link to this message
-
Re: Multi threaded queue consumers
Hi Vamsi

A short update, it appeared that Using lockPath() (from Apache curator)
hurts the throuhput a lot.
I wrote to the relevant mailing list a description of the issue.

best regards
Benjamin
On Mon, Nov 25, 2013 at 8:43 AM, Vamsi Devaki <[EMAIL PROTECTED]>wrote:

> On Sun, Nov 24, 2013 at 9:57 PM, Sznajder ForMailingList
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > I share with you a very problematic (for me...) issue, I encountered.
> >
> > I wrote a very simple Queue (Producer and consumer).
> >
> > The producer, put to the ZK Queue items (integer) when the queue reached
> > half of its initial size.
> > And Consumer just picks from the list and prints to a log file the
> consumed
> > item and sleeps some time.
> >
> > In addition, I defined the Consumer to be multi-threaded
> >
> > I tested the following configurations on different quorums (when each
> server
> > is 16 cores)
> >
> >  Quorum of 2 servers : One consumer and one producer, 2 threads for the
> > consumer:
> > In 5 minutes, I consumed 1320 items.
> >
> > When I make the consumer to be 16 threads, I get: 10890 items consumed.
> All
> > is OK.
> >
> >
> >
> >
> > However, when I move to a quorum of three, with 2 consumers (and each
> > consumer 16 threads), I suddenly get:
> > In 5 minutes 12634 items consumed. That means, that adding a new consumer
> > did not enhance the throughput of the system.
> >
> >
> > I am blocked with this issue, several days, and any help will be
> welcome!!
> >
> > Benjamin
> >
>
> Hello Benjamin,
>
> Did you measure add/delete thru-put for your cluster? If not, can you
> check if it is significantly higher than 12k?
>
> This should help to get you started:
> https://github.com/brownsys/zookeeper-benchmark
>
> Regards,
> Vamsi
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB