Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> Multi threaded queue consumers


+
Sznajder ForMailingList 2013-11-24, 16:27
Copy link to this message
-
Re: Multi threaded queue consumers
On Sun, Nov 24, 2013 at 9:57 PM, Sznajder ForMailingList
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> I share with you a very problematic (for me...) issue, I encountered.
>
> I wrote a very simple Queue (Producer and consumer).
>
> The producer, put to the ZK Queue items (integer) when the queue reached
> half of its initial size.
> And Consumer just picks from the list and prints to a log file the consumed
> item and sleeps some time.
>
> In addition, I defined the Consumer to be multi-threaded
>
> I tested the following configurations on different quorums (when each server
> is 16 cores)
>
>  Quorum of 2 servers : One consumer and one producer, 2 threads for the
> consumer:
> In 5 minutes, I consumed 1320 items.
>
> When I make the consumer to be 16 threads, I get: 10890 items consumed. All
> is OK.
>
>
>
>
> However, when I move to a quorum of three, with 2 consumers (and each
> consumer 16 threads), I suddenly get:
> In 5 minutes 12634 items consumed. That means, that adding a new consumer
> did not enhance the throughput of the system.
>
>
> I am blocked with this issue, several days, and any help will be welcome!!
>
> Benjamin
>

Hello Benjamin,

Did you measure add/delete thru-put for your cluster? If not, can you
check if it is significantly higher than 12k?

This should help to get you started:
https://github.com/brownsys/zookeeper-benchmark

Regards,
Vamsi
+
Sznajder ForMailingList 2013-11-28, 14:34
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB