Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> sync vs. async vs. multi performances


+
N Keywal 2012-02-14, 16:00
+
Flavio Junqueira 2012-02-14, 16:11
+
N Keywal 2012-02-14, 16:16
+
Camille Fournier 2012-02-14, 17:24
+
Flavio Junqueira 2012-02-14, 17:42
+
Camille Fournier 2012-02-14, 18:38
+
Ted Dunning 2012-02-14, 19:00
+
N Keywal 2012-02-14, 20:37
+
Ariel Weisberg 2012-02-14, 20:48
+
Flavio Junqueira 2012-02-14, 21:02
+
Ariel Weisberg 2012-02-14, 23:18
+
Ted Dunning 2012-02-15, 04:57
+
Flavio Junqueira 2012-02-17, 08:41
+
Ariel Weisberg 2012-02-18, 15:17
+
Flavio Junqueira 2012-02-19, 11:04
Copy link to this message
-
Re: sync vs. async vs. multi performances
Just as an arithmetic check, hundreds x 20 ms = seconds, not minutes.  Even
1000 x 0.02 s = 20 s which isn't all that long.  Faster is nice, but this
doesn't reach "minutes".  And as Flavio points out, if the recovery is
threaded, it will be faster.

On Tue, Feb 14, 2012 at 3:37 PM, N Keywal <[EMAIL PROTECTED]> wrote:

> Hi,
>
> Thanks for the replies.
>
> It's used when assigning the regions (kind of dataset) to the regionserver
> (jvm process in a physical server). There is one zookeeper node per region.
> On a server failure, there is typically a few hundreds regions to reassign,
> with multiple status written in . On paper, if we need 0,02s per node, that
> makes it to the minute to recover, just for zookeeper.
>
> That's theory. I haven't done a precise measurement yet.
>
>
> Anyway, if ZooKeeper can be faster, it's always very interesting :-)
>
>
> Cheers,
>
> N.
>
>
> On Tue, Feb 14, 2012 at 8:00 PM, Ted Dunning <[EMAIL PROTECTED]>
> wrote:
>
> > These results are about what is expected although the might be a little
> > more extreme.
> >
> > I doubt very much that hbase is mutating zk nodes fast enough for this to
> > matter much.
> >
> > Sent from my iPhone
> >
> > On Feb 14, 2012, at 8:00, N Keywal <[EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > I've done a test with Zookeeper 3.4.2 to compare the performances of
> > > synchronous vs. asynchronous vs. multi when creating znode (variations
> > > around:
> > > calling 10000 times zk.create("/dummyTest", "dummy".getBytes(),
> > > ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);) The code is at
> the
> > > end of the mail.
> > >
> > > I've tested different environments:
> > > - 1 linux server with the client and 1 zookeeper node on the same
> machine
> > > - 1 linux server for the client, 1 for 1 zookeeper node.
> > > - 6 linux servers, 1 for the client, 5 for 5 zookeeper nodes.
> > >
> > > Server are middle range, with 4*2 cores, jdk 1.6. ZK was on its own HD.
> > >
> > > But the results are comparable:
> > >
> > > Using the sync API, it takes 200 seconds for 10K creations, so around
> > 0.02
> > > second per call.
> > > Using the async API, it takes 2 seconds for 10K (including waiting for
> > the
> > > last callback message)
> > > Using the "multi" available since 3.4, it takes less than 1 second,
> again
> > > for 10K.
> > >
> > > I'm surprised by the time taken by the sync operation, I was not
> > expecting
> > > it to be that slow. The gap between async & sync is quite huge.
> > >
> > > Is this something expected? Zookeeper is used in critical functions in
> > > Hadoop/Hbase, I was looking at the possible benefits of using "multi",
> > but
> > > it seems low compared to async (well ~3 times faster :-). There are
> many
> > > small data creations/deletions with the sync API in the existing hbase
> > > algorithms, it would not be simple to replace them all by asynchronous
> > > calls...
> > >
> > > Cheers,
> > >
> > > N.
> > >
> > > --
> > >
> > > public class ZookeeperTest {
> > >  static ZooKeeper zk;
> > >  static int nbTests = 10000;
> > >
> > >  private ZookeeperTest() {
> > >  }
> > >
> > >  public static void test11() throws Exception {
> > >    for (int i = 0; i < nbTests; ++i) {
> > >      zk.create("/dummyTest_" + i, "dummy".getBytes(),
> > > ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
> > >    }
> > >  }
> > >
> > >
> > >  public static void test51() throws Exception {
> > >    final AtomicInteger counter = new AtomicInteger(0);
> > >
> > >    for (int i = 0; i < nbTests; ++i) {
> > >      zk.create("/dummyTest_" + i, "dummy".getBytes(),
> > > ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT,
> > >        new AsyncCallback.StringCallback() {
> > >          public void processResult(int i, String s, Object o, String
> s1)
> > {
> > >            counter.incrementAndGet();
> > >          }
> > >        }
> > >        , null);
> > >    }
> > >
> > >    while (counter.get() != nbTests) {
> > >      Thread.sleep(1);
> > >    }
> > >  }
>