Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> Question about sharing Zookeeper connections


+
YUNG-LIN HO 2011-02-18, 09:24
+
Ted Dunning 2011-02-18, 17:31
+
Benjamin Reed 2011-02-18, 17:58
+
Peco Karayanev 2011-02-18, 19:13
+
Ted Dunning 2011-02-18, 19:40
Copy link to this message
-
Re: Question about sharing Zookeeper connections
Sorry to resurrect an old discussion, but this is pertinent to me.

So, just to clarify, what you're saying is - if you've got a subsystem in an
app that uses ZK then you should be fine (i.e both from a thread-safety
perspective as well as a performance perspective) sharing a single
connection between all the threads running code in that subsystem (this is
what I interpreted your phrase "kind of use" to mean) - even if, say, you
expect thousands of calls to happen on that one connection over the span of,
say, a couple of minutes?

On the other hand are you also saying that it is advisable to have separate
ZK connections for each subystem that requires it? Even if they are all
running in the same JVM?

Ishaaq

On 19 February 2011 06:40, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Most applications just share a single connection for all uses of a single
> class.  That might be viewed at the logically simplest extreme of a simple
> connection pool.
>
> Why did you need more than one connection for each kind of use?
>
> On Fri, Feb 18, 2011 at 11:13 AM, Peco Karayanev <[EMAIL PROTECTED]
> >wrote:
>
> > Hi,
> > I slightly disagree with the priority for a "connection pool" for
> > zookeeper.
> > I had to implement a connection pool/reuse for a smaller environment
> (under
> > 50 nodes). I have a toolchain that can be composed of parts, and each
> part
> > needed some synchronization through zookeeper. So even on a smaller
> system
> > these heavier toolchains started using a lot of physical sessions and
> > connections. Hence the connection pool. Also from performance perspective
> > establishing a TCP connection for every session adds latency overhead.
> >
> > Cheers
> > Peco
> >
> >
> > On Fri, Feb 18, 2011 at 11:58 AM, Benjamin Reed <[EMAIL PROTECTED]>
> > wrote:
> >
> > > it does seem like a good idea to make multiple zk handles share a
> > > connection, but as ted points out, they may have different timeouts,
> > which
> > > would make the sharing logic quite complicated. i think the
> > implementation
> > > might also be quite complicated. having said that, if someone could
> come
> > up
> > > with a simple and correct connection sharing implementation, we (or at
> > > least
> > > i) would be open to it.
> > >
> > > ben
> > >
> > > On Fri, Feb 18, 2011 at 9:31 AM, Ted Dunning <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > On Fri, Feb 18, 2011 at 1:24 AM, YUNG-LIN HO <[EMAIL PROTECTED]>
> > wrote:
> > > >
> > > > > Because zookeeper clients will try to keep session alive by sending
> a
> > > > ping
> > > > > request every 2 seconds. If libraries in an application do not
> share
> > > > > connections with each other, they would flood the zookeeper server
> > with
> > > > > unnecessary requests and drag down performance of the server.
> > > > >
> > > >
> > > > Make sure that you have a valid reason to worry first.
> > > >
> > > > Do you have thousands of clients?
> > > >
> > > > If not, these keep-alives are likely to be undetectable, load-wise.
> > > >
> > > >
> > > > > I am wondering is there any connection manager exists in the
> > > > > Hadoop/Zookeeper project that helps users to share connections?
> > > > >
> > > >
> > > > Yes.  Zookeeper.
> > > >
> > > > Just open a single connection and pass it around via a singleton of
> > some
> > > > kind or your favorite dependency injection technique.
> > > >
> > > > This isn't always a great idea since your disconnect and expiration
> > > > strategies might differ between different uses in important ways.
> > > >
> > >
> >
>
+
Thomas Koch 2011-02-19, 15:18