Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper, mail # user - Multi threaded queue consumers


Copy link to this message
-
Multi threaded queue consumers
Sznajder ForMailingList 2013-11-24, 16:27
Hi,

I share with you a very problematic (for me...) issue, I encountered.

I wrote a very simple Queue (Producer and consumer).

The producer, put to the ZK Queue items (integer) when the queue reached
half of its initial size.
And Consumer just picks from the list and prints to a log file the consumed
item and sleeps some time.

In addition, I defined the Consumer to be multi-threaded

I tested the following configurations on different quorums (when each
server is 16 cores)

 Quorum of 2 servers : One consumer and one producer, 2 threads for the
consumer:
In 5 minutes, I consumed 1320 items.

When I make the consumer to be 16 threads, I get: 10890 items consumed. All
is OK.
However, when I move to a quorum of three, with 2 consumers (and each
consumer 16 threads), I suddenly get:
In 5 minutes 12634 items consumed. That means, that adding a new consumer
did not enhance the throughput of the system.
I am blocked with this issue, several days, and any help will be welcome!!

Benjamin