Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> sync vs. async vs. multi performances


+
N Keywal 2012-02-14, 16:00
+
Flavio Junqueira 2012-02-14, 16:11
+
N Keywal 2012-02-14, 16:16
+
Camille Fournier 2012-02-14, 17:24
+
Flavio Junqueira 2012-02-14, 17:42
Copy link to this message
-
Re: sync vs. async vs. multi performances
True. We do have an outstanding jira that Ben filed on a perf problem in
3.4.X that could be contributing:
https://issues.apache.org/jira/browse/ZOOKEEPER-1390
On Tue, Feb 14, 2012 at 12:42 PM, Flavio Junqueira <[EMAIL PROTECTED]>wrote:

> If I'm reading it correctly, these tests are getting 20ms per op. This is
> too high. I think we were getting something like 5ms before.
>
> -Flavio
>
> On Feb 14, 2012, at 6:24 PM, Camille Fournier wrote:
>
> > Sync calls have to make a complete roundtrip before the next call from
> that
> > client will happen. It's not surprising at all that it would take quite a
> > bit longer to do a sync call than an async call. It could be that the
> > bottleneck in this case is your client, not your server. If the sync
> calls
> > are happening amongst clients on many different servers, it probably
> > doesn't matter.
> >
> > C
> >
> > On Tue, Feb 14, 2012 at 11:00 AM, N Keywal <[EMAIL PROTECTED]> wrote:
> >
> >> Hi,
> >>
> >> I've done a test with Zookeeper 3.4.2 to compare the performances of
> >> synchronous vs. asynchronous vs. multi when creating znode (variations
> >> around:
> >> calling 10000 times zk.create("/dummyTest", "dummy".getBytes(),
> >> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);) The code is at the
> >> end of the mail.
> >>
> >> I've tested different environments:
> >> - 1 linux server with the client and 1 zookeeper node on the same
> machine
> >> - 1 linux server for the client, 1 for 1 zookeeper node.
> >> - 6 linux servers, 1 for the client, 5 for 5 zookeeper nodes.
> >>
> >> Server are middle range, with 4*2 cores, jdk 1.6. ZK was on its own HD.
> >>
> >> But the results are comparable:
> >>
> >> Using the sync API, it takes 200 seconds for 10K creations, so around
> 0.02
> >> second per call.
> >> Using the async API, it takes 2 seconds for 10K (including waiting for
> the
> >> last callback message)
> >> Using the "multi" available since 3.4, it takes less than 1 second,
> again
> >> for 10K.
> >>
> >> I'm surprised by the time taken by the sync operation, I was not
> expecting
> >> it to be that slow. The gap between async & sync is quite huge.
> >>
> >> Is this something expected? Zookeeper is used in critical functions in
> >> Hadoop/Hbase, I was looking at the possible benefits of using "multi",
> but
> >> it seems low compared to async (well ~3 times faster :-). There are many
> >> small data creations/deletions with the sync API in the existing hbase
> >> algorithms, it would not be simple to replace them all by asynchronous
> >> calls...
> >>
> >> Cheers,
> >>
> >> N.
> >>
> >> --
> >>
> >> public class ZookeeperTest {
> >> static ZooKeeper zk;
> >> static int nbTests = 10000;
> >>
> >> private ZookeeperTest() {
> >> }
> >>
> >> public static void test11() throws Exception {
> >>   for (int i = 0; i < nbTests; ++i) {
> >>     zk.create("/dummyTest_" + i, "dummy".getBytes(),
> >> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
> >>   }
> >> }
> >>
> >>
> >> public static void test51() throws Exception {
> >>   final AtomicInteger counter = new AtomicInteger(0);
> >>
> >>   for (int i = 0; i < nbTests; ++i) {
> >>     zk.create("/dummyTest_" + i, "dummy".getBytes(),
> >> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT,
> >>       new AsyncCallback.StringCallback() {
> >>         public void processResult(int i, String s, Object o, String s1)
> {
> >>           counter.incrementAndGet();
> >>         }
> >>       }
> >>       , null);
> >>   }
> >>
> >>   while (counter.get() != nbTests) {
> >>     Thread.sleep(1);
> >>   }
> >> }
> >>
> >> public static void test41() throws Exception {
> >>   ArrayList<Op> ops = new ArrayList<Op>(nbTests);
> >>   for (int i = 0; i < nbTests; ++i) {
> >>     ops.add(
> >>       Op.create("/dummyTest_" + i, "dummy".getBytes(),
> >> ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT)
> >>     );
> >>   }
> >>
> >>   zk.multi(ops);
> >> }
> >>
> >> public static void delete() throws Exception {
> >>   ArrayList<Op> ops = new ArrayList<Op>(nbTests);
+
Ted Dunning 2012-02-14, 19:00
+
N Keywal 2012-02-14, 20:37
+
Ariel Weisberg 2012-02-14, 20:48
+
Flavio Junqueira 2012-02-14, 21:02
+
Ariel Weisberg 2012-02-14, 23:18
+
Ted Dunning 2012-02-15, 04:57
+
Flavio Junqueira 2012-02-17, 08:41
+
Ariel Weisberg 2012-02-18, 15:17
+
Flavio Junqueira 2012-02-19, 11:04
+
Ted Dunning 2012-02-15, 04:54
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB