Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> sync vs. async vs. multi performances


+
N Keywal 2012-02-14, 16:00
Copy link to this message
-
Re: sync vs. async vs. multi performances
From your description, it sounds like you're using one disk device for the ZooKeeper server. Is it correct?

-Flavio

On Feb 14, 2012, at 5:00 PM, N Keywal wrote:

> Hi,
>
> I've done a test with Zookeeper 3.4.2 to compare the performances of
> synchronous vs. asynchronous vs. multi when creating znode (variations
> around:
> calling 10000 times zk.create("/dummyTest", "dummy".getBytes(),
> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);) The code is at the
> end of the mail.
>
> I've tested different environments:
> - 1 linux server with the client and 1 zookeeper node on the same machine
> - 1 linux server for the client, 1 for 1 zookeeper node.
> - 6 linux servers, 1 for the client, 5 for 5 zookeeper nodes.
>
> Server are middle range, with 4*2 cores, jdk 1.6. ZK was on its own HD.
>
> But the results are comparable:
>
> Using the sync API, it takes 200 seconds for 10K creations, so around 0.02
> second per call.
> Using the async API, it takes 2 seconds for 10K (including waiting for the
> last callback message)
> Using the "multi" available since 3.4, it takes less than 1 second, again
> for 10K.
>
> I'm surprised by the time taken by the sync operation, I was not expecting
> it to be that slow. The gap between async & sync is quite huge.
>
> Is this something expected? Zookeeper is used in critical functions in
> Hadoop/Hbase, I was looking at the possible benefits of using "multi", but
> it seems low compared to async (well ~3 times faster :-). There are many
> small data creations/deletions with the sync API in the existing hbase
> algorithms, it would not be simple to replace them all by asynchronous
> calls...
>
> Cheers,
>
> N.
>
> --
>
> public class ZookeeperTest {
>  static ZooKeeper zk;
>  static int nbTests = 10000;
>
>  private ZookeeperTest() {
>  }
>
>  public static void test11() throws Exception {
>    for (int i = 0; i < nbTests; ++i) {
>      zk.create("/dummyTest_" + i, "dummy".getBytes(),
> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
>    }
>  }
>
>
>  public static void test51() throws Exception {
>    final AtomicInteger counter = new AtomicInteger(0);
>
>    for (int i = 0; i < nbTests; ++i) {
>      zk.create("/dummyTest_" + i, "dummy".getBytes(),
> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT,
>        new AsyncCallback.StringCallback() {
>          public void processResult(int i, String s, Object o, String s1) {
>            counter.incrementAndGet();
>          }
>        }
>        , null);
>    }
>
>    while (counter.get() != nbTests) {
>      Thread.sleep(1);
>    }
>  }
>
>  public static void test41() throws Exception {
>    ArrayList<Op> ops = new ArrayList<Op>(nbTests);
>    for (int i = 0; i < nbTests; ++i) {
>      ops.add(
>        Op.create("/dummyTest_" + i, "dummy".getBytes(),
> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT)
>      );
>    }
>
>    zk.multi(ops);
>  }
>
>  public static void delete() throws Exception {
>    ArrayList<Op> ops = new ArrayList<Op>(nbTests);
>
>    for (int i = 0; i < nbTests; ++i) {
>      ops.add(
>        Op.delete("/dummyTest_" + i,-1)
>      );
>    }
>
>    zk.multi(ops);
>  }
>
>
>  public static void test(String connection, String testName) throws
> Throwable{
>    Method m = ZookeeperTest.class.getMethod(testName);
>
>    zk = new ZooKeeper(connection, 20000, new Watcher() {
>      public void process(WatchedEvent watchedEvent) {
>      }
>    });
>
>    final long start = System.currentTimeMillis();
>
>    try {
>      m.invoke(null);
>    } catch (IllegalAccessException e) {
>      throw e;
>    } catch (InvocationTargetException e) {
>      throw e.getTargetException();
>    }
>
>    final long end = System.currentTimeMillis();
>
>    zk.close();
>
>    final long endClose = System.currentTimeMillis();
>
>    System.out.println(testName+":  ExeTime= " + (end - start) );
>  }
>
>  public static void main(String... args) throws Throwable {
>      test(args[0], args[1]);
>  }
> }

flavio
junqueira
 
research scientist
 
[EMAIL PROTECTED]
direct +34 93-183-8828
 
avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301
+
N Keywal 2012-02-14, 16:16
+
Camille Fournier 2012-02-14, 17:24
+
Flavio Junqueira 2012-02-14, 17:42
+
Camille Fournier 2012-02-14, 18:38
+
Ted Dunning 2012-02-14, 19:00
+
N Keywal 2012-02-14, 20:37
+
Ariel Weisberg 2012-02-14, 20:48
+
Flavio Junqueira 2012-02-14, 21:02
+
Ariel Weisberg 2012-02-14, 23:18
+
Ted Dunning 2012-02-15, 04:57
+
Flavio Junqueira 2012-02-17, 08:41
+
Ariel Weisberg 2012-02-18, 15:17
+
Flavio Junqueira 2012-02-19, 11:04
+
Ted Dunning 2012-02-15, 04:54