Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - The Jenkins VMs are increasingly slow / overloaded


Copy link to this message
-
Re: The Jenkins VMs are increasingly slow / overloaded
Andrew Purtell 2013-03-27, 14:18
In general moving from using the m1.large (2 vcores, 7.5 GB RAM) to the
m1.xlarge (4 vcores, 15 GB RAM) instance type for the slaves helped with a
build/test timeout, so now I'd about claim the test environment is sane. We
are now seeing that replication tests are flapping, occasionally timing out
internally:

See
http://54.241.6.143/job/HBase-0.94/org.apache.hbase$hbase/24/testReport/junit/org.apache.hadoop.hbase.replication/TestReplicationQueueFailoverCompressed/queueFailover/
and
http://54.241.6.143/job/HBase-0.94-Security/org.apache.hbase$hbase/7/testReport/junit/org.apache.hadoop.hbase.replication/TestReplicationQueueFailover/queueFailover/
The 0.94 and 0.94-security builds are alternating between green and red as
a result.

Perhaps we should reopen/revisit either adjusting the internal timeouts for
these tests or the other JIRA about moving minicluster replication tests to
hbase-it.
On Wed, Mar 27, 2013 at 1:49 AM, Nick Dimiduk <[EMAIL PROTECTED]> wrote:

> On Tue, Mar 26, 2013 at 1:28 PM, Andrew Purtell <[EMAIL PROTECTED]>
> wrote:
>
> > The HBase 0.94 build is now testing green!
> > http://54.241.6.143/job/HBase-0.94/
> >
>
> ^5!
>
> On Tue, Mar 26, 2013 at 1:47 AM, Andrew Purtell <[EMAIL PROTECTED]>
> wrote:
> >
> > > I found that Maven was being killed on the slaves by the Linux OOM
> killer
> > > sometimes for >= 0.95. Seems the m1.large process didn't have enough
> > memory
> > > to host the Jenkins slave, Maven with its 3G+ heap, and the forked JVMs
> > for
> > > the medium and large tests at the same time. Switching to the m1.xlarge
> > > type resolved this. Now the 0.95 and trunk builds fail for what looks
> > like
> > > a legitimate problem with a hanging test.
> > >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>

--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)