Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> sync vs. async vs. multi performances


+
N Keywal 2012-02-14, 16:00
+
Flavio Junqueira 2012-02-14, 16:11
+
N Keywal 2012-02-14, 16:16
+
Camille Fournier 2012-02-14, 17:24
+
Flavio Junqueira 2012-02-14, 17:42
Copy link to this message
-
Re: sync vs. async vs. multi performances
True. We do have an outstanding jira that Ben filed on a perf problem in
3.4.X that could be contributing:
https://issues.apache.org/jira/browse/ZOOKEEPER-1390
On Tue, Feb 14, 2012 at 12:42 PM, Flavio Junqueira <[EMAIL PROTECTED]>wrote:

> If I'm reading it correctly, these tests are getting 20ms per op. This is
> too high. I think we were getting something like 5ms before.
>
> -Flavio
>
> On Feb 14, 2012, at 6:24 PM, Camille Fournier wrote:
>
> > Sync calls have to make a complete roundtrip before the next call from
> that
> > client will happen. It's not surprising at all that it would take quite a
> > bit longer to do a sync call than an async call. It could be that the
> > bottleneck in this case is your client, not your server. If the sync
> calls
> > are happening amongst clients on many different servers, it probably
> > doesn't matter.
> >
> > C
> >
> > On Tue, Feb 14, 2012 at 11:00 AM, N Keywal <[EMAIL PROTECTED]> wrote:
> >
> >> Hi,
> >>
> >> I've done a test with Zookeeper 3.4.2 to compare the performances of
> >> synchronous vs. asynchronous vs. multi when creating znode (variations
> >> around:
> >> calling 10000 times zk.create("/dummyTest", "dummy".getBytes(),
> >> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);) The code is at the
> >> end of the mail.
> >>
> >> I've tested different environments:
> >> - 1 linux server with the client and 1 zookeeper node on the same
> machine
> >> - 1 linux server for the client, 1 for 1 zookeeper node.
> >> - 6 linux servers, 1 for the client, 5 for 5 zookeeper nodes.
> >>
> >> Server are middle range, with 4*2 cores, jdk 1.6. ZK was on its own HD.
> >>
> >> But the results are comparable:
> >>
> >> Using the sync API, it takes 200 seconds for 10K creations, so around
> 0.02
> >> second per call.
> >> Using the async API, it takes 2 seconds for 10K (including waiting for
> the
> >> last callback message)
> >> Using the "multi" available since 3.4, it takes less than 1 second,
> again
> >> for 10K.
> >>
> >> I'm surprised by the time taken by the sync operation, I was not
> expecting
> >> it to be that slow. The gap between async & sync is quite huge.
> >>
> >> Is this something expected? Zookeeper is used in critical functions in
> >> Hadoop/Hbase, I was looking at the possible benefits of using "multi",
> but
> >> it seems low compared to async (well ~3 times faster :-). There are many
> >> small data creations/deletions with the sync API in the existing hbase
> >> algorithms, it would not be simple to replace them all by asynchronous
> >> calls...
> >>
> >> Cheers,
> >>
> >> N.
> >>
> >> --
> >>
> >> public class ZookeeperTest {
> >> static ZooKeeper zk;
> >> static int nbTests = 10000;
> >>
> >> private ZookeeperTest() {
> >> }
> >>
> >> public static void test11() throws Exception {
> >>   for (int i = 0; i < nbTests; ++i) {
> >>     zk.create("/dummyTest_" + i, "dummy".getBytes(),
> >> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
> >>   }
> >> }
> >>
> >>
> >> public static void test51() throws Exception {
> >>   final AtomicInteger counter = new AtomicInteger(0);
> >>
> >>   for (int i = 0; i < nbTests; ++i) {
> >>     zk.create("/dummyTest_" + i, "dummy".getBytes(),
> >> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT,
> >>       new AsyncCallback.StringCallback() {
> >>         public void processResult(int i, String s, Object o, String s1)
> {
> >>           counter.incrementAndGet();
> >>         }
> >>       }
> >>       , null);
> >>   }
> >>
> >>   while (counter.get() != nbTests) {
> >>     Thread.sleep(1);
> >>   }
> >> }
> >>
> >> public static void test41() throws Exception {
> >>   ArrayList<Op> ops = new ArrayList<Op>(nbTests);
> >>   for (int i = 0; i < nbTests; ++i) {
> >>     ops.add(
> >>       Op.create("/dummyTest_" + i, "dummy".getBytes(),
> >> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT)
> >>     );
> >>   }
> >>
> >>   zk.multi(ops);
> >> }
> >>
> >> public static void delete() throws Exception {
> >>   ArrayList<Op> ops = new ArrayList<Op>(nbTests);
+
Ted Dunning 2012-02-14, 19:00
+
N Keywal 2012-02-14, 20:37
+
Ariel Weisberg 2012-02-14, 20:48
+
Flavio Junqueira 2012-02-14, 21:02
+
Ariel Weisberg 2012-02-14, 23:18
+
Ted Dunning 2012-02-15, 04:57
+
Flavio Junqueira 2012-02-17, 08:41
+
Ariel Weisberg 2012-02-18, 15:17
+
Flavio Junqueira 2012-02-19, 11:04
+
Ted Dunning 2012-02-15, 04:54