Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> sync vs. async vs. multi performances


Copy link to this message
-
Re: sync vs. async vs. multi performances
Sync calls have to make a complete roundtrip before the next call from that
client will happen. It's not surprising at all that it would take quite a
bit longer to do a sync call than an async call. It could be that the
bottleneck in this case is your client, not your server. If the sync calls
are happening amongst clients on many different servers, it probably
doesn't matter.

C

On Tue, Feb 14, 2012 at 11:00 AM, N Keywal <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I've done a test with Zookeeper 3.4.2 to compare the performances of
> synchronous vs. asynchronous vs. multi when creating znode (variations
> around:
> calling 10000 times zk.create("/dummyTest", "dummy".getBytes(),
> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);) The code is at the
> end of the mail.
>
> I've tested different environments:
> - 1 linux server with the client and 1 zookeeper node on the same machine
> - 1 linux server for the client, 1 for 1 zookeeper node.
> - 6 linux servers, 1 for the client, 5 for 5 zookeeper nodes.
>
> Server are middle range, with 4*2 cores, jdk 1.6. ZK was on its own HD.
>
> But the results are comparable:
>
> Using the sync API, it takes 200 seconds for 10K creations, so around 0.02
> second per call.
> Using the async API, it takes 2 seconds for 10K (including waiting for the
> last callback message)
> Using the "multi" available since 3.4, it takes less than 1 second, again
> for 10K.
>
> I'm surprised by the time taken by the sync operation, I was not expecting
> it to be that slow. The gap between async & sync is quite huge.
>
> Is this something expected? Zookeeper is used in critical functions in
> Hadoop/Hbase, I was looking at the possible benefits of using "multi", but
> it seems low compared to async (well ~3 times faster :-). There are many
> small data creations/deletions with the sync API in the existing hbase
> algorithms, it would not be simple to replace them all by asynchronous
> calls...
>
> Cheers,
>
> N.
>
> --
>
> public class ZookeeperTest {
>  static ZooKeeper zk;
>  static int nbTests = 10000;
>
>  private ZookeeperTest() {
>  }
>
>  public static void test11() throws Exception {
>    for (int i = 0; i < nbTests; ++i) {
>      zk.create("/dummyTest_" + i, "dummy".getBytes(),
> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
>    }
>  }
>
>
>  public static void test51() throws Exception {
>    final AtomicInteger counter = new AtomicInteger(0);
>
>    for (int i = 0; i < nbTests; ++i) {
>      zk.create("/dummyTest_" + i, "dummy".getBytes(),
> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT,
>        new AsyncCallback.StringCallback() {
>          public void processResult(int i, String s, Object o, String s1) {
>            counter.incrementAndGet();
>          }
>        }
>        , null);
>    }
>
>    while (counter.get() != nbTests) {
>      Thread.sleep(1);
>    }
>  }
>
>  public static void test41() throws Exception {
>    ArrayList<Op> ops = new ArrayList<Op>(nbTests);
>    for (int i = 0; i < nbTests; ++i) {
>      ops.add(
>        Op.create("/dummyTest_" + i, "dummy".getBytes(),
> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT)
>      );
>    }
>
>    zk.multi(ops);
>  }
>
>  public static void delete() throws Exception {
>    ArrayList<Op> ops = new ArrayList<Op>(nbTests);
>
>    for (int i = 0; i < nbTests; ++i) {
>      ops.add(
>        Op.delete("/dummyTest_" + i,-1)
>      );
>    }
>
>    zk.multi(ops);
>  }
>
>
>  public static void test(String connection, String testName) throws
> Throwable{
>    Method m = ZookeeperTest.class.getMethod(testName);
>
>    zk = new ZooKeeper(connection, 20000, new Watcher() {
>      public void process(WatchedEvent watchedEvent) {
>      }
>    });
>
>    final long start = System.currentTimeMillis();
>
>    try {
>      m.invoke(null);
>    } catch (IllegalAccessException e) {
>      throw e;
>    } catch (InvocationTargetException e) {
>      throw e.getTargetException();
>    }
>
>    final long end = System.currentTimeMillis();
>