Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> The Jenkins VMs are increasingly slow / overloaded


+
Andrew Purtell 2013-03-19, 15:49
+
Nick Dimiduk 2013-03-19, 16:23
+
Andrew Purtell 2013-03-19, 17:48
+
Stack 2013-03-19, 19:00
+
Andrew Purtell 2013-03-19, 23:30
+
lars hofhansl 2013-03-20, 06:33
+
Andrew Purtell 2013-03-20, 08:57
+
Andrew Purtell 2013-03-20, 21:37
+
Ted Yu 2013-03-20, 21:39
+
Andrew Purtell 2013-03-20, 21:44
+
Andrew Purtell 2013-03-20, 21:44
+
Nick Dimiduk 2013-03-19, 20:50
+
Stack 2013-03-22, 04:38
+
Stack 2013-03-23, 04:51
+
Andrew Purtell 2013-03-23, 10:11
+
Stack 2013-03-23, 20:24
+
Andrew Purtell 2013-03-24, 05:56
+
Andrew Purtell 2013-03-26, 00:47
+
Andrew Purtell 2013-03-26, 20:28
+
Nick Dimiduk 2013-03-27, 00:49
+
Andrew Purtell 2013-03-27, 14:18
+
Ted Yu 2013-03-27, 14:38
Copy link to this message
-
Re: The Jenkins VMs are increasingly slow / overloaded
True, but unlike 0.94 the state of 0.95 and trunk is impacted by Stack's
wrangling with Maven to find a sane site and assembly, a number of build
failures are due to that. Also you'll note that prior to yesterday the
Linux OOM killer was nuking the bloated Maven processes on the build
slaves. Let's give these builds a bit of time for this stuff to get sorted
out. The failures in 0.94 seem immediately actionable.
On Wed, Mar 27, 2013 at 3:38 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Trunk and 0.95 builds are not in good shape.
> 0.95 builds have been failing for 32 times.
>
> On Apache Jenkins, looks like TestAssignmentManagerOnCluster has failed
> quite often for 0.95 and trunk builds.
>
> On Wed, Mar 27, 2013 at 7:18 AM, Andrew Purtell <[EMAIL PROTECTED]>
> wrote:
>
> > In general moving from using the m1.large (2 vcores, 7.5 GB RAM) to the
> > m1.xlarge (4 vcores, 15 GB RAM) instance type for the slaves helped with
> a
> > build/test timeout, so now I'd about claim the test environment is sane.
> We
> > are now seeing that replication tests are flapping, occasionally timing
> out
> > internally:
> >
> > See
> >
> >
> http://54.241.6.143/job/HBase-0.94/org.apache.hbase$hbase/24/testReport/junit/org.apache.hadoop.hbase.replication/TestReplicationQueueFailoverCompressed/queueFailover/
> >
> >
> > and
> >
> >
> http://54.241.6.143/job/HBase-0.94-Security/org.apache.hbase$hbase/7/testReport/junit/org.apache.hadoop.hbase.replication/TestReplicationQueueFailover/queueFailover/
> >
> >
> > The 0.94 and 0.94-security builds are alternating between green and red
> as
> > a result.
> >
> > Perhaps we should reopen/revisit either adjusting the internal timeouts
> for
> > these tests or the other JIRA about moving minicluster replication tests
> to
> > hbase-it.
> >
> >
> > On Wed, Mar 27, 2013 at 1:49 AM, Nick Dimiduk <[EMAIL PROTECTED]>
> wrote:
> >
> > > On Tue, Mar 26, 2013 at 1:28 PM, Andrew Purtell <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > The HBase 0.94 build is now testing green!
> > > > http://54.241.6.143/job/HBase-0.94/
> > > >
> > >
> > > ^5!
> > >
> > > On Tue, Mar 26, 2013 at 1:47 AM, Andrew Purtell <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > I found that Maven was being killed on the slaves by the Linux OOM
> > > killer
> > > > > sometimes for >= 0.95. Seems the m1.large process didn't have
> enough
> > > > memory
> > > > > to host the Jenkins slave, Maven with its 3G+ heap, and the forked
> > JVMs
> > > > for
> > > > > the medium and large tests at the same time. Switching to the
> > m1.xlarge
> > > > > type resolved this. Now the 0.95 and trunk builds fail for what
> looks
> > > > like
> > > > > a legitimate problem with a hanging test.
> > > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > >    - Andy
> > > >
> > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > > > (via Tom White)
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>

--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
+
Ted Yu 2013-03-27, 20:19
+
Ted Yu 2013-04-05, 23:39
+
Andrew Purtell 2013-04-05, 23:44
+
Andrew Purtell 2013-04-05, 23:48
+
Stack 2013-03-26, 21:59
+
Andrew Purtell 2013-03-26, 22:14
+
Andrew Purtell 2013-04-18, 19:46
+
Ted Yu 2013-04-19, 00:22
+
Andrew Purtell 2013-04-19, 00:25
+
Ted Yu 2013-04-19, 00:29
+
Andrew Purtell 2013-04-19, 00:34
+
Andrew Purtell 2013-06-10, 18:14
+
Stack 2013-06-11, 00:03
+
Elliott Clark 2013-06-10, 19:42
+
Andrew Purtell 2013-06-10, 20:50
+
Andrew Purtell 2013-07-15, 22:27
+
Ted Yu 2013-07-15, 22:30
+
Andrew Purtell 2013-07-15, 22:37