Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper, mail # user - sync vs. async vs. multi performances


+
N Keywal 2012-02-14, 16:00
+
Flavio Junqueira 2012-02-14, 16:11
+
N Keywal 2012-02-14, 16:16
+
Camille Fournier 2012-02-14, 17:24
Copy link to this message
-
Re: sync vs. async vs. multi performances
Flavio Junqueira 2012-02-14, 17:42
If I'm reading it correctly, these tests are getting 20ms per op. This is too high. I think we were getting something like 5ms before.

-Flavio

On Feb 14, 2012, at 6:24 PM, Camille Fournier wrote:

> Sync calls have to make a complete roundtrip before the next call from that
> client will happen. It's not surprising at all that it would take quite a
> bit longer to do a sync call than an async call. It could be that the
> bottleneck in this case is your client, not your server. If the sync calls
> are happening amongst clients on many different servers, it probably
> doesn't matter.
>
> C
>
> On Tue, Feb 14, 2012 at 11:00 AM, N Keywal <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> I've done a test with Zookeeper 3.4.2 to compare the performances of
>> synchronous vs. asynchronous vs. multi when creating znode (variations
>> around:
>> calling 10000 times zk.create("/dummyTest", "dummy".getBytes(),
>> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);) The code is at the
>> end of the mail.
>>
>> I've tested different environments:
>> - 1 linux server with the client and 1 zookeeper node on the same machine
>> - 1 linux server for the client, 1 for 1 zookeeper node.
>> - 6 linux servers, 1 for the client, 5 for 5 zookeeper nodes.
>>
>> Server are middle range, with 4*2 cores, jdk 1.6. ZK was on its own HD.
>>
>> But the results are comparable:
>>
>> Using the sync API, it takes 200 seconds for 10K creations, so around 0.02
>> second per call.
>> Using the async API, it takes 2 seconds for 10K (including waiting for the
>> last callback message)
>> Using the "multi" available since 3.4, it takes less than 1 second, again
>> for 10K.
>>
>> I'm surprised by the time taken by the sync operation, I was not expecting
>> it to be that slow. The gap between async & sync is quite huge.
>>
>> Is this something expected? Zookeeper is used in critical functions in
>> Hadoop/Hbase, I was looking at the possible benefits of using "multi", but
>> it seems low compared to async (well ~3 times faster :-). There are many
>> small data creations/deletions with the sync API in the existing hbase
>> algorithms, it would not be simple to replace them all by asynchronous
>> calls...
>>
>> Cheers,
>>
>> N.
>>
>> --
>>
>> public class ZookeeperTest {
>> static ZooKeeper zk;
>> static int nbTests = 10000;
>>
>> private ZookeeperTest() {
>> }
>>
>> public static void test11() throws Exception {
>>   for (int i = 0; i < nbTests; ++i) {
>>     zk.create("/dummyTest_" + i, "dummy".getBytes(),
>> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
>>   }
>> }
>>
>>
>> public static void test51() throws Exception {
>>   final AtomicInteger counter = new AtomicInteger(0);
>>
>>   for (int i = 0; i < nbTests; ++i) {
>>     zk.create("/dummyTest_" + i, "dummy".getBytes(),
>> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT,
>>       new AsyncCallback.StringCallback() {
>>         public void processResult(int i, String s, Object o, String s1) {
>>           counter.incrementAndGet();
>>         }
>>       }
>>       , null);
>>   }
>>
>>   while (counter.get() != nbTests) {
>>     Thread.sleep(1);
>>   }
>> }
>>
>> public static void test41() throws Exception {
>>   ArrayList<Op> ops = new ArrayList<Op>(nbTests);
>>   for (int i = 0; i < nbTests; ++i) {
>>     ops.add(
>>       Op.create("/dummyTest_" + i, "dummy".getBytes(),
>> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT)
>>     );
>>   }
>>
>>   zk.multi(ops);
>> }
>>
>> public static void delete() throws Exception {
>>   ArrayList<Op> ops = new ArrayList<Op>(nbTests);
>>
>>   for (int i = 0; i < nbTests; ++i) {
>>     ops.add(
>>       Op.delete("/dummyTest_" + i,-1)
>>     );
>>   }
>>
>>   zk.multi(ops);
>> }
>>
>>
>> public static void test(String connection, String testName) throws
>> Throwable{
>>   Method m = ZookeeperTest.class.getMethod(testName);
>>
>>   zk = new ZooKeeper(connection, 20000, new Watcher() {
>>     public void process(WatchedEvent watchedEvent) {
>>     }

flavio
junqueira
 
research scientist
 
[EMAIL PROTECTED]
direct +34 93-183-8828
 
avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301
+
Camille Fournier 2012-02-14, 18:38
+
Ted Dunning 2012-02-14, 19:00
+
N Keywal 2012-02-14, 20:37
+
Ariel Weisberg 2012-02-14, 20:48
+
Flavio Junqueira 2012-02-14, 21:02
+
Ariel Weisberg 2012-02-14, 23:18
+
Ted Dunning 2012-02-15, 04:57
+
Flavio Junqueira 2012-02-17, 08:41
+
Ariel Weisberg 2012-02-18, 15:17
+
Flavio Junqueira 2012-02-19, 11:04
+
Ted Dunning 2012-02-15, 04:54