Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> sync vs. async vs. multi performances


+
N Keywal 2012-02-14, 16:00
+
Flavio Junqueira 2012-02-14, 16:11
+
N Keywal 2012-02-14, 16:16
+
Camille Fournier 2012-02-14, 17:24
Copy link to this message
-
Re: sync vs. async vs. multi performances
If I'm reading it correctly, these tests are getting 20ms per op. This is too high. I think we were getting something like 5ms before.

-Flavio

On Feb 14, 2012, at 6:24 PM, Camille Fournier wrote:

> Sync calls have to make a complete roundtrip before the next call from that
> client will happen. It's not surprising at all that it would take quite a
> bit longer to do a sync call than an async call. It could be that the
> bottleneck in this case is your client, not your server. If the sync calls
> are happening amongst clients on many different servers, it probably
> doesn't matter.
>
> C
>
> On Tue, Feb 14, 2012 at 11:00 AM, N Keywal <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> I've done a test with Zookeeper 3.4.2 to compare the performances of
>> synchronous vs. asynchronous vs. multi when creating znode (variations
>> around:
>> calling 10000 times zk.create("/dummyTest", "dummy".getBytes(),
>> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);) The code is at the
>> end of the mail.
>>
>> I've tested different environments:
>> - 1 linux server with the client and 1 zookeeper node on the same machine
>> - 1 linux server for the client, 1 for 1 zookeeper node.
>> - 6 linux servers, 1 for the client, 5 for 5 zookeeper nodes.
>>
>> Server are middle range, with 4*2 cores, jdk 1.6. ZK was on its own HD.
>>
>> But the results are comparable:
>>
>> Using the sync API, it takes 200 seconds for 10K creations, so around 0.02
>> second per call.
>> Using the async API, it takes 2 seconds for 10K (including waiting for the
>> last callback message)
>> Using the "multi" available since 3.4, it takes less than 1 second, again
>> for 10K.
>>
>> I'm surprised by the time taken by the sync operation, I was not expecting
>> it to be that slow. The gap between async & sync is quite huge.
>>
>> Is this something expected? Zookeeper is used in critical functions in
>> Hadoop/Hbase, I was looking at the possible benefits of using "multi", but
>> it seems low compared to async (well ~3 times faster :-). There are many
>> small data creations/deletions with the sync API in the existing hbase
>> algorithms, it would not be simple to replace them all by asynchronous
>> calls...
>>
>> Cheers,
>>
>> N.
>>
>> --
>>
>> public class ZookeeperTest {
>> static ZooKeeper zk;
>> static int nbTests = 10000;
>>
>> private ZookeeperTest() {
>> }
>>
>> public static void test11() throws Exception {
>>   for (int i = 0; i < nbTests; ++i) {
>>     zk.create("/dummyTest_" + i, "dummy".getBytes(),
>> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
>>   }
>> }
>>
>>
>> public static void test51() throws Exception {
>>   final AtomicInteger counter = new AtomicInteger(0);
>>
>>   for (int i = 0; i < nbTests; ++i) {
>>     zk.create("/dummyTest_" + i, "dummy".getBytes(),
>> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT,
>>       new AsyncCallback.StringCallback() {
>>         public void processResult(int i, String s, Object o, String s1) {
>>           counter.incrementAndGet();
>>         }
>>       }
>>       , null);
>>   }
>>
>>   while (counter.get() != nbTests) {
>>     Thread.sleep(1);
>>   }
>> }
>>
>> public static void test41() throws Exception {
>>   ArrayList<Op> ops = new ArrayList<Op>(nbTests);
>>   for (int i = 0; i < nbTests; ++i) {
>>     ops.add(
>>       Op.create("/dummyTest_" + i, "dummy".getBytes(),
>> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT)
>>     );
>>   }
>>
>>   zk.multi(ops);
>> }
>>
>> public static void delete() throws Exception {
>>   ArrayList<Op> ops = new ArrayList<Op>(nbTests);
>>
>>   for (int i = 0; i < nbTests; ++i) {
>>     ops.add(
>>       Op.delete("/dummyTest_" + i,-1)
>>     );
>>   }
>>
>>   zk.multi(ops);
>> }
>>
>>
>> public static void test(String connection, String testName) throws
>> Throwable{
>>   Method m = ZookeeperTest.class.getMethod(testName);
>>
>>   zk = new ZooKeeper(connection, 20000, new Watcher() {
>>     public void process(WatchedEvent watchedEvent) {
>>     }

flavio
junqueira
 
research scientist
 
[EMAIL PROTECTED]
direct +34 93-183-8828
 
avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301
+
Camille Fournier 2012-02-14, 18:38
+
Ted Dunning 2012-02-14, 19:00
+
N Keywal 2012-02-14, 20:37
+
Ariel Weisberg 2012-02-14, 20:48
+
Flavio Junqueira 2012-02-14, 21:02
+
Ariel Weisberg 2012-02-14, 23:18
+
Ted Dunning 2012-02-15, 04:57
+
Flavio Junqueira 2012-02-17, 08:41
+
Ariel Weisberg 2012-02-18, 15:17
+
Flavio Junqueira 2012-02-19, 11:04
+
Ted Dunning 2012-02-15, 04:54
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB