Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> Multi threaded queue consumers


Copy link to this message
-
Re: Multi threaded queue consumers
Hi Vamsi

A short update, it appeared that Using lockPath() (from Apache curator)
hurts the throuhput a lot.
I wrote to the relevant mailing list a description of the issue.

best regards
Benjamin
On Mon, Nov 25, 2013 at 8:43 AM, Vamsi Devaki <[EMAIL PROTECTED]>wrote:

> On Sun, Nov 24, 2013 at 9:57 PM, Sznajder ForMailingList
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > I share with you a very problematic (for me...) issue, I encountered.
> >
> > I wrote a very simple Queue (Producer and consumer).
> >
> > The producer, put to the ZK Queue items (integer) when the queue reached
> > half of its initial size.
> > And Consumer just picks from the list and prints to a log file the
> consumed
> > item and sleeps some time.
> >
> > In addition, I defined the Consumer to be multi-threaded
> >
> > I tested the following configurations on different quorums (when each
> server
> > is 16 cores)
> >
> >  Quorum of 2 servers : One consumer and one producer, 2 threads for the
> > consumer:
> > In 5 minutes, I consumed 1320 items.
> >
> > When I make the consumer to be 16 threads, I get: 10890 items consumed.
> All
> > is OK.
> >
> >
> >
> >
> > However, when I move to a quorum of three, with 2 consumers (and each
> > consumer 16 threads), I suddenly get:
> > In 5 minutes 12634 items consumed. That means, that adding a new consumer
> > did not enhance the throughput of the system.
> >
> >
> > I am blocked with this issue, several days, and any help will be
> welcome!!
> >
> > Benjamin
> >
>
> Hello Benjamin,
>
> Did you measure add/delete thru-put for your cluster? If not, can you
> check if it is significantly higher than 12k?
>
> This should help to get you started:
> https://github.com/brownsys/zookeeper-benchmark
>
> Regards,
> Vamsi
>