-Re: Multi threaded queue consumers
Sznajder ForMailingList 2013-11-28, 14:34
A short update, it appeared that Using lockPath() (from Apache curator)
hurts the throuhput a lot.
I wrote to the relevant mailing list a description of the issue.
On Mon, Nov 25, 2013 at 8:43 AM, Vamsi Devaki <[EMAIL PROTECTED]>wrote:
> On Sun, Nov 24, 2013 at 9:57 PM, Sznajder ForMailingList
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> > I share with you a very problematic (for me...) issue, I encountered.
> > I wrote a very simple Queue (Producer and consumer).
> > The producer, put to the ZK Queue items (integer) when the queue reached
> > half of its initial size.
> > And Consumer just picks from the list and prints to a log file the
> > item and sleeps some time.
> > In addition, I defined the Consumer to be multi-threaded
> > I tested the following configurations on different quorums (when each
> > is 16 cores)
> > Quorum of 2 servers : One consumer and one producer, 2 threads for the
> > consumer:
> > In 5 minutes, I consumed 1320 items.
> > When I make the consumer to be 16 threads, I get: 10890 items consumed.
> > is OK.
> > However, when I move to a quorum of three, with 2 consumers (and each
> > consumer 16 threads), I suddenly get:
> > In 5 minutes 12634 items consumed. That means, that adding a new consumer
> > did not enhance the throughput of the system.
> > I am blocked with this issue, several days, and any help will be
> > Benjamin
> Hello Benjamin,
> Did you measure add/delete thru-put for your cluster? If not, can you
> check if it is significantly higher than 12k?
> This should help to get you started: