Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> Serious problem processing hearbeat on login stampede


+
Chang Song 2011-04-13, 13:35
+
Patrick Hunt 2011-04-13, 15:21
+
Chang Song 2011-04-13, 22:02
+
Patrick Hunt 2011-04-14, 01:30
+
Patrick Hunt 2011-04-14, 04:53
+
Ted Dunning 2011-04-14, 05:24
+
Chang Song 2011-04-14, 11:54
+
Chang Song 2011-04-14, 14:03
+
Mahadev Konar 2011-04-14, 16:00
+
Chang Song 2011-04-14, 21:58
+
Chang Song 2011-04-14, 14:03
+
Patrick Hunt 2011-04-14, 16:04
+
Benjamin Reed 2011-04-14, 17:59
+
Chang Song 2011-04-14, 22:10
+
Benjamin Reed 2011-04-14, 22:16
+
Chang Song 2011-04-14, 22:20
+
Ted Dunning 2011-04-14, 22:31
+
Ted Dunning 2011-04-14, 22:34
+
Chang Song 2011-04-16, 03:51
+
Ted Dunning 2011-04-16, 05:21
+
Chang Song 2011-04-16, 06:25
+
Ted Dunning 2011-04-16, 20:43
+
Chang Song 2011-04-17, 06:52
+
Ted Dunning 2011-04-16, 20:46
+
Chang Song 2011-04-17, 07:48
+
Chang Song 2011-04-14, 22:02
+
Lakshman 2011-04-16, 11:36
+
Chang Song 2011-04-16, 12:30
+
Fournier, Camille F. [Tec... 2011-04-18, 14:49
+
Ted Dunning 2011-04-18, 16:03
+
Mahadev Konar 2011-04-18, 18:16
+
Chang Song 2011-04-19, 10:26
+
Ted Dunning 2011-04-19, 16:59
Copy link to this message
-
Re: Serious problem processing hearbeat on login stampede
As a note, I believe we just used this patch to solve a major issue we were
seeing.  We were having problems when power to a node was pulled, and thus
hung tcp sessions on the servers.  With many connections, each close
operation was taking 2 seconds and held up the server significantly enough
to start incorrectly closing other sessions.  By disabling linger, these
hanging sessions were closed immediately and the problem went away.

Thanks Chang!
~Jared

On Tue, Apr 19, 2011 at 10:59 AM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Where is this set?
>
> Why does this cause this problem?
>
> 2011/4/19 Chang Song <[EMAIL PROTECTED]>
>
> >
> > Problem solved.
> > it was socket linger option set to 2 sec timeout.
> >
> > We have verified that the original problem goes away when we turn off
> > linger option.
> > No longer a mystery ;)
> >
> >
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1049
> >
> >
> > Chang
> >
> >
> > 2011. 4. 19., 오전 3:16, Mahadev Konar 작성:
> >
> > > Camille, Ted,
> > > Can we continue the discussion on
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-1049?
> > >
> > > We should track all the suggestions/issues on the jira.
> > >
> > > thanks
> > > mahadev
> > >
> > > On Mon, Apr 18, 2011 at 9:03 AM, Ted Dunning <[EMAIL PROTECTED]>
> > wrote:
> > >> Interesting.  It does seem to suggestion the session expiration is
> > >> expensive.
> > >>
> > >> There is a concurrent table in guava that provides very good
> > multi-threaded
> > >> performance.  I think that is achieved by using a number of locks and
> > then
> > >> distributing threads across the locks according to the hash slot being
> > used.
> > >>  But I would have expected any in memory operation to complete very
> > quickly.
> > >>
> > >> Is it possible that the locks on the session table are held longer
> than
> > they
> > >> should be?
> > >>
> > >> 2011/4/18 Fournier, Camille F. [Tech] <[EMAIL PROTECTED]>
> > >>
> > >>> Is it possible this is related to this report back in February?
> > >>>
> > >>>
> >
> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201102.mbox/%[EMAIL PROTECTED]%3E
> > >>>
> > >>> I theorized that the issue might be due to synchronization on the
> > session
> > >>> table, but never got enough information to finish the investigation.
> > >>>
> > >>
> > >
> > >
> > >
> > > --
> > > thanks
> > > mahadev
> > > @mahadevkonar
> >
> >
>
+
Ted Dunning 2011-07-01, 16:03
+
Chang Song 2011-07-02, 05:22
+
Patrick Hunt 2011-07-05, 18:04
+
Chang Song 2011-07-06, 05:02