Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> Multi threaded queue consumers

Copy link to this message
Re: Multi threaded queue consumers
On Sun, Nov 24, 2013 at 9:57 PM, Sznajder ForMailingList
> Hi,
> I share with you a very problematic (for me...) issue, I encountered.
> I wrote a very simple Queue (Producer and consumer).
> The producer, put to the ZK Queue items (integer) when the queue reached
> half of its initial size.
> And Consumer just picks from the list and prints to a log file the consumed
> item and sleeps some time.
> In addition, I defined the Consumer to be multi-threaded
> I tested the following configurations on different quorums (when each server
> is 16 cores)
>  Quorum of 2 servers : One consumer and one producer, 2 threads for the
> consumer:
> In 5 minutes, I consumed 1320 items.
> When I make the consumer to be 16 threads, I get: 10890 items consumed. All
> is OK.
> However, when I move to a quorum of three, with 2 consumers (and each
> consumer 16 threads), I suddenly get:
> In 5 minutes 12634 items consumed. That means, that adding a new consumer
> did not enhance the throughput of the system.
> I am blocked with this issue, several days, and any help will be welcome!!
> Benjamin

Hello Benjamin,

Did you measure add/delete thru-put for your cluster? If not, can you
check if it is significantly higher than 12k?

This should help to get you started: