Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper, mail # user - Multi threaded queue consumers


Copy link to this message
-
Re: Multi threaded queue consumers
Vamsi Devaki 2013-11-25, 06:43
On Sun, Nov 24, 2013 at 9:57 PM, Sznajder ForMailingList
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> I share with you a very problematic (for me...) issue, I encountered.
>
> I wrote a very simple Queue (Producer and consumer).
>
> The producer, put to the ZK Queue items (integer) when the queue reached
> half of its initial size.
> And Consumer just picks from the list and prints to a log file the consumed
> item and sleeps some time.
>
> In addition, I defined the Consumer to be multi-threaded
>
> I tested the following configurations on different quorums (when each server
> is 16 cores)
>
>  Quorum of 2 servers : One consumer and one producer, 2 threads for the
> consumer:
> In 5 minutes, I consumed 1320 items.
>
> When I make the consumer to be 16 threads, I get: 10890 items consumed. All
> is OK.
>
>
>
>
> However, when I move to a quorum of three, with 2 consumers (and each
> consumer 16 threads), I suddenly get:
> In 5 minutes 12634 items consumed. That means, that adding a new consumer
> did not enhance the throughput of the system.
>
>
> I am blocked with this issue, several days, and any help will be welcome!!
>
> Benjamin
>

Hello Benjamin,

Did you measure add/delete thru-put for your cluster? If not, can you
check if it is significantly higher than 12k?

This should help to get you started:
https://github.com/brownsys/zookeeper-benchmark

Regards,
Vamsi