Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> Serious problem processing hearbeat on login stampede


+
Chang Song 2011-04-13, 13:35
+
Patrick Hunt 2011-04-13, 15:21
+
Chang Song 2011-04-13, 22:02
+
Patrick Hunt 2011-04-14, 01:30
+
Patrick Hunt 2011-04-14, 04:53
+
Ted Dunning 2011-04-14, 05:24
+
Chang Song 2011-04-14, 11:54
+
Chang Song 2011-04-14, 14:03
+
Mahadev Konar 2011-04-14, 16:00
+
Chang Song 2011-04-14, 21:58
+
Chang Song 2011-04-14, 14:03
+
Patrick Hunt 2011-04-14, 16:04
+
Benjamin Reed 2011-04-14, 17:59
+
Chang Song 2011-04-14, 22:10
+
Benjamin Reed 2011-04-14, 22:16
+
Chang Song 2011-04-14, 22:20
+
Ted Dunning 2011-04-14, 22:31
+
Ted Dunning 2011-04-14, 22:34
+
Chang Song 2011-04-16, 03:51
+
Ted Dunning 2011-04-16, 05:21
+
Chang Song 2011-04-16, 06:25
+
Ted Dunning 2011-04-16, 20:43
+
Chang Song 2011-04-17, 06:52
+
Ted Dunning 2011-04-16, 20:46
+
Chang Song 2011-04-17, 07:48
+
Chang Song 2011-04-14, 22:02
+
Lakshman 2011-04-16, 11:36
+
Chang Song 2011-04-16, 12:30
+
Fournier, Camille F. [Tec... 2011-04-18, 14:49
+
Ted Dunning 2011-04-18, 16:03
+
Mahadev Konar 2011-04-18, 18:16
+
Chang Song 2011-04-19, 10:26
+
Ted Dunning 2011-04-19, 16:59
Copy link to this message
-
Re: Serious problem processing hearbeat on login stampede
As a note, I believe we just used this patch to solve a major issue we were
seeing.  We were having problems when power to a node was pulled, and thus
hung tcp sessions on the servers.  With many connections, each close
operation was taking 2 seconds and held up the server significantly enough
to start incorrectly closing other sessions.  By disabling linger, these
hanging sessions were closed immediately and the problem went away.

Thanks Chang!
~Jared

On Tue, Apr 19, 2011 at 10:59 AM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Where is this set?
>
> Why does this cause this problem?
>
> 2011/4/19 Chang Song <[EMAIL PROTECTED]>
>
> >
> > Problem solved.
> > it was socket linger option set to 2 sec timeout.
> >
> > We have verified that the original problem goes away when we turn off
> > linger option.
> > No longer a mystery ;)
> >
> >
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1049
> >
> >
> > Chang
> >
> >
> > 2011. 4. 19., 오전 3:16, Mahadev Konar 작성:
> >
> > > Camille, Ted,
> > > Can we continue the discussion on
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-1049?
> > >
> > > We should track all the suggestions/issues on the jira.
> > >
> > > thanks
> > > mahadev
> > >
> > > On Mon, Apr 18, 2011 at 9:03 AM, Ted Dunning <[EMAIL PROTECTED]>
> > wrote:
> > >> Interesting.  It does seem to suggestion the session expiration is
> > >> expensive.
> > >>
> > >> There is a concurrent table in guava that provides very good
> > multi-threaded
> > >> performance.  I think that is achieved by using a number of locks and
> > then
> > >> distributing threads across the locks according to the hash slot being
> > used.
> > >>  But I would have expected any in memory operation to complete very
> > quickly.
> > >>
> > >> Is it possible that the locks on the session table are held longer
> than
> > they
> > >> should be?
> > >>
> > >> 2011/4/18 Fournier, Camille F. [Tech] <[EMAIL PROTECTED]>
> > >>
> > >>> Is it possible this is related to this report back in February?
> > >>>
> > >>>
> >
> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201102.mbox/%[EMAIL PROTECTED]%3E
> > >>>
> > >>> I theorized that the issue might be due to synchronization on the
> > session
> > >>> table, but never got enough information to finish the investigation.
> > >>>
> > >>
> > >
> > >
> > >
> > > --
> > > thanks
> > > mahadev
> > > @mahadevkonar
> >
> >
>
+
Ted Dunning 2011-07-01, 16:03
+
Chang Song 2011-07-02, 05:22
+
Patrick Hunt 2011-07-05, 18:04
+
Chang Song 2011-07-06, 05:02
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB