I've noticed that TestHLog is currently hanging in trunk (haven't checked
other branches).  Oddly the tests actually complete, but then the test hangs
in teardown.

Seems to be something in the server shutdown hooks.  git bisect tracks down
the hang to this commit:

commit 9c195c7ef350a932a9901a2069b96694d202c675
Author: Michael Stack <[EMAIL PROTECTED]>
Date:   Fri Sep 30 21:45:20 2011 +0000

    HBASE-4209 The HBase hbase-daemon.sh SIGKILLs master when stopping it

    git-svn-id:
https://svn.apache.org/repos/asf/hbase/trunk@117784913f79535-47bb-0310-9956-ffa450edef68
Anyone else noticed this on TestHLog or other tests?  I think it may be
behind some of our odd test cleanup issues up in Jenkins.

--gh

Somehow TestHLog was never actually shutting down the mini-cluster?

The following change lets the test exit successfully:

diff --git
a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
index 663b318..13f821c 100644
--- a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
+++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
@@ -54,6 +54,7 @@ import
org.apache.hadoop.hdfs.server.namenode.LeaseManager;
 import org.apache.hadoop.io.SequenceFile;
 import org.apache.log4j.Level;
 import org.junit.After;
+import org.junit.AfterClass;
 import org.junit.Before;
 import org.junit.BeforeClass;
 import org.junit.Test;
@@ -120,6 +121,11 @@ public class TestHLog  {
     oldLogDir = new Path(hbaseDir, ".oldlogs");
     dir = new Path(hbaseDir, getName());
   }
+  @AfterClass
+  public static void tearDownAfterClass() throws Exception {
+    TEST_UTIL.shutdownMiniCluster();
+  }
+
   private static String getName() {
     // TODO Auto-generated method stub
     return "TestHLog";
On Wed, Oct 5, 2011 at 11:23 AM, Gary Helmling <[EMAIL PROTECTED]> wrote:

> I've noticed that TestHLog is currently hanging in trunk (haven't checked
> other branches).  Oddly the tests actually complete, but then the test hangs
> in teardown.
>
> Seems to be something in the server shutdown hooks.  git bisect tracks down
> the hang to this commit:
>
> commit 9c195c7ef350a932a9901a2069b96694d202c675
> Author: Michael Stack <[EMAIL PROTECTED]>
> Date:   Fri Sep 30 21:45:20 2011 +0000
>
>     HBASE-4209 The HBase hbase-daemon.sh SIGKILLs master when stopping it
>
>     git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@117784913f79535-47bb-0310-9956-ffa450edef68
>
>
> Anyone else noticed this on TestHLog or other tests?  I think it may be
> behind some of our odd test cleanup issues up in Jenkins.
>
> --gh
>
>

Nice catch! Hadn't had a chance to look into that test yet, but I feel this
kind of 'dirty' testing is (unfortunately) going to be pretty prevalent.
Going to work on a deep dive on a bunch of tests soon though.

-Jesse Yates

On Wed, Oct 5, 2011 at 11:55 AM, Gary Helmling <[EMAIL PROTECTED]> wrote:

> Somehow TestHLog was never actually shutting down the mini-cluster?
>
> The following change lets the test exit successfully:
>
> diff --git
> a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> index 663b318..13f821c 100644
> --- a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> @@ -54,6 +54,7 @@ import
> org.apache.hadoop.hdfs.server.namenode.LeaseManager;
>  import org.apache.hadoop.io.SequenceFile;
>  import org.apache.log4j.Level;
>  import org.junit.After;
> +import org.junit.AfterClass;
>  import org.junit.Before;
>  import org.junit.BeforeClass;
>  import org.junit.Test;
> @@ -120,6 +121,11 @@ public class TestHLog  {
>     oldLogDir = new Path(hbaseDir, ".oldlogs");
>     dir = new Path(hbaseDir, getName());
>   }
> +  @AfterClass
> +  public static void tearDownAfterClass() throws Exception {
> +    TEST_UTIL.shutdownMiniCluster();
> +  }
> +
>   private static String getName() {
>     // TODO Auto-generated method stub
>     return "TestHLog";
>
>
> On Wed, Oct 5, 2011 at 11:23 AM, Gary Helmling <[EMAIL PROTECTED]>
> wrote:
>
> > I've noticed that TestHLog is currently hanging in trunk (haven't checked
> > other branches).  Oddly the tests actually complete, but then the test
> hangs
> > in teardown.
> >
> > Seems to be something in the server shutdown hooks.  git bisect tracks
> down
> > the hang to this commit:
> >
> > commit 9c195c7ef350a932a9901a2069b96694d202c675
> > Author: Michael Stack <[EMAIL PROTECTED]>
> > Date:   Fri Sep 30 21:45:20 2011 +0000
> >
> >     HBASE-4209 The HBase hbase-daemon.sh SIGKILLs master when stopping it
> >
> >     git-svn-id:
> https://svn.apache.org/repos/asf/hbase/trunk@117784913f79535-47bb-0310-9956-ffa450edef68
> >
> >
> > Anyone else noticed this on TestHLog or other tests?  I think it may be
> > behind some of our odd test cleanup issues up in Jenkins.
> >
> > --gh
> >
> >
>

+CC Roman who worked on the patch identified by the bisect.

Roman, does Gary's analysis make sense to you?

-Todd

On Wed, Oct 5, 2011 at 11:55 AM, Gary Helmling <[EMAIL PROTECTED]> wrote:
> Somehow TestHLog was never actually shutting down the mini-cluster?
>
> The following change lets the test exit successfully:
>
> diff --git
> a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> index 663b318..13f821c 100644
> --- a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> +++ b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> @@ -54,6 +54,7 @@ import
> org.apache.hadoop.hdfs.server.namenode.LeaseManager;
>  import org.apache.hadoop.io.SequenceFile;
>  import org.apache.log4j.Level;
>  import org.junit.After;
> +import org.junit.AfterClass;
>  import org.junit.Before;
>  import org.junit.BeforeClass;
>  import org.junit.Test;
> @@ -120,6 +121,11 @@ public class TestHLog  {
>     oldLogDir = new Path(hbaseDir, ".oldlogs");
>     dir = new Path(hbaseDir, getName());
>   }
> +  @AfterClass
> +  public static void tearDownAfterClass() throws Exception {
> +    TEST_UTIL.shutdownMiniCluster();
> +  }
> +
>   private static String getName() {
>     // TODO Auto-generated method stub
>     return "TestHLog";
>
>
> On Wed, Oct 5, 2011 at 11:23 AM, Gary Helmling <[EMAIL PROTECTED]> wrote:
>
>> I've noticed that TestHLog is currently hanging in trunk (haven't checked
>> other branches).  Oddly the tests actually complete, but then the test hangs
>> in teardown.
>>
>> Seems to be something in the server shutdown hooks.  git bisect tracks down
>> the hang to this commit:
>>
>> commit 9c195c7ef350a932a9901a2069b96694d202c675
>> Author: Michael Stack <[EMAIL PROTECTED]>
>> Date:   Fri Sep 30 21:45:20 2011 +0000
>>
>>     HBASE-4209 The HBase hbase-daemon.sh SIGKILLs master when stopping it
>>
>>     git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@117784913f79535-47bb-0310-9956-ffa450edef68
>>
>>
>> Anyone else noticed this on TestHLog or other tests?  I think it may be
>> behind some of our odd test cleanup issues up in Jenkins.
>>
>> --gh
>>
>>
>

--
Todd Lipcon
Software Engineer, Cloudera

Something else seems to be going on.  With the call to shutdownMiniCluster()
the first run of TestHLog passes.  But when I try running in a loop, the
second run always seems to hang.

Thread dump here: http://pastebin.com/f18Wfa3T
On Wed, Oct 5, 2011 at 12:00 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> +CC Roman who worked on the patch identified by the bisect.
>
> Roman, does Gary's analysis make sense to you?
>
> -Todd
>
> On Wed, Oct 5, 2011 at 11:55 AM, Gary Helmling <[EMAIL PROTECTED]>
> wrote:
> > Somehow TestHLog was never actually shutting down the mini-cluster?
> >
> > The following change lets the test exit successfully:
> >
> > diff --git
> > a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> > b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> > index 663b318..13f821c 100644
> > ---
> a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> > +++
> b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> > @@ -54,6 +54,7 @@ import
> > org.apache.hadoop.hdfs.server.namenode.LeaseManager;
> >  import org.apache.hadoop.io.SequenceFile;
> >  import org.apache.log4j.Level;
> >  import org.junit.After;
> > +import org.junit.AfterClass;
> >  import org.junit.Before;
> >  import org.junit.BeforeClass;
> >  import org.junit.Test;
> > @@ -120,6 +121,11 @@ public class TestHLog  {
> >     oldLogDir = new Path(hbaseDir, ".oldlogs");
> >     dir = new Path(hbaseDir, getName());
> >   }
> > +  @AfterClass
> > +  public static void tearDownAfterClass() throws Exception {
> > +    TEST_UTIL.shutdownMiniCluster();
> > +  }
> > +
> >   private static String getName() {
> >     // TODO Auto-generated method stub
> >     return "TestHLog";
> >
> >
> > On Wed, Oct 5, 2011 at 11:23 AM, Gary Helmling <[EMAIL PROTECTED]>
> wrote:
> >
> >> I've noticed that TestHLog is currently hanging in trunk (haven't
> checked
> >> other branches).  Oddly the tests actually complete, but then the test
> hangs
> >> in teardown.
> >>
> >> Seems to be something in the server shutdown hooks.  git bisect tracks
> down
> >> the hang to this commit:
> >>
> >> commit 9c195c7ef350a932a9901a2069b96694d202c675
> >> Author: Michael Stack <[EMAIL PROTECTED]>
> >> Date:   Fri Sep 30 21:45:20 2011 +0000
> >>
> >>     HBASE-4209 The HBase hbase-daemon.sh SIGKILLs master when stopping
> it
> >>
> >>     git-svn-id:
> https://svn.apache.org/repos/asf/hbase/trunk@117784913f79535-47bb-0310-9956-ffa450edef68
> >>
> >>
> >> Anyone else noticed this on TestHLog or other tests?  I think it may be
> >> behind some of our odd test cleanup issues up in Jenkins.
> >>
> >> --gh
> >>
> >>
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Easy to repro Gary?  I just add in your above patch and then loop?
The first run completely goes down?  The regionserver is waiting on
all regions to close before it can go down (and then the master go
down).
St.Ack

On Wed, Oct 5, 2011 at 12:13 PM, Gary Helmling <[EMAIL PROTECTED]> wrote:
> Something else seems to be going on.  With the call to shutdownMiniCluster()
> the first run of TestHLog passes.  But when I try running in a loop, the
> second run always seems to hang.
>
> Thread dump here: http://pastebin.com/f18Wfa3T
>
>
> On Wed, Oct 5, 2011 at 12:00 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> +CC Roman who worked on the patch identified by the bisect.
>>
>> Roman, does Gary's analysis make sense to you?
>>
>> -Todd
>>
>> On Wed, Oct 5, 2011 at 11:55 AM, Gary Helmling <[EMAIL PROTECTED]>
>> wrote:
>> > Somehow TestHLog was never actually shutting down the mini-cluster?
>> >
>> > The following change lets the test exit successfully:
>> >
>> > diff --git
>> > a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
>> > b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
>> > index 663b318..13f821c 100644
>> > ---
>> a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
>> > +++
>> b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
>> > @@ -54,6 +54,7 @@ import
>> > org.apache.hadoop.hdfs.server.namenode.LeaseManager;
>> >  import org.apache.hadoop.io.SequenceFile;
>> >  import org.apache.log4j.Level;
>> >  import org.junit.After;
>> > +import org.junit.AfterClass;
>> >  import org.junit.Before;
>> >  import org.junit.BeforeClass;
>> >  import org.junit.Test;
>> > @@ -120,6 +121,11 @@ public class TestHLog  {
>> >     oldLogDir = new Path(hbaseDir, ".oldlogs");
>> >     dir = new Path(hbaseDir, getName());
>> >   }
>> > +  @AfterClass
>> > +  public static void tearDownAfterClass() throws Exception {
>> > +    TEST_UTIL.shutdownMiniCluster();
>> > +  }
>> > +
>> >   private static String getName() {
>> >     // TODO Auto-generated method stub
>> >     return "TestHLog";
>> >
>> >
>> > On Wed, Oct 5, 2011 at 11:23 AM, Gary Helmling <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> I've noticed that TestHLog is currently hanging in trunk (haven't
>> checked
>> >> other branches).  Oddly the tests actually complete, but then the test
>> hangs
>> >> in teardown.
>> >>
>> >> Seems to be something in the server shutdown hooks.  git bisect tracks
>> down
>> >> the hang to this commit:
>> >>
>> >> commit 9c195c7ef350a932a9901a2069b96694d202c675
>> >> Author: Michael Stack <[EMAIL PROTECTED]>
>> >> Date:   Fri Sep 30 21:45:20 2011 +0000
>> >>
>> >>     HBASE-4209 The HBase hbase-daemon.sh SIGKILLs master when stopping
>> it
>> >>
>> >>     git-svn-id:
>> https://svn.apache.org/repos/asf/hbase/trunk@117784913f79535-47bb-0310-9956-ffa450edef68
>> >>
>> >>
>> >> Anyone else noticed this on TestHLog or other tests?  I think it may be
>> >> behind some of our odd test cleanup issues up in Jenkins.
>> >>
>> >> --gh
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>

On Wed, Oct 5, 2011 at 3:05 PM, Stack <[EMAIL PROTECTED]> wrote:

> Easy to repro Gary?  I just add in your above patch and then loop?
> The first run completely goes down?  The regionserver is waiting on
> all regions to close before it can go down (and then the master go
> down).
> St.Ack
>
This patch isn't sufficient, unfortunately.  It seemed to work against the
HBASE-4209 commit, but clustter shutdown still hangs on current trunk.
Still waiting for regionservers to go down.

Without the patch you get the shutdown hook thread waiting to join the RS's:

"main" prio=10 tid=0x00000000403f1800 nid=0x2701 in Object.wait()
[0x00007f34067c8000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000abf5b438> (a
org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread)
    at java.lang.Thread.join(Thread.java:1186)
    - locked <0x00000000abf5b438> (a
org.apache.hadoop.hbase.regionserver.ShutdownHook$ShutdownHookThread)
    at java.lang.Thread.join(Thread.java:1239)
    at
java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
With the patch it just shifts this to JVMClusterUtil.shutdown() waiting to
join the RS threads:

"main" prio=10 tid=0x00000000401ec800 nid=0x153 in Object.wait()
[0x00007fb76a918000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000acc60aa8> (a
org.apache.hadoop.hbase.util.JVMClusterUtil$RegionServerThread)
    at java.lang.Thread.join(Thread.java:1186)
    - locked <0x00000000acc60aa8> (a
org.apache.hadoop.hbase.util.JVMClusterUtil$RegionServerThread)
    at java.lang.Thread.join(Thread.java:1239)
    at
org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:230)
Any idea what's causing the region closing to hang?  The surefire plugin
doesn't seem to write the TestHLog-output.txt file if the test is hanging.
Any way to force it to do so?  Kind of useless.

>
> On Wed, Oct 5, 2011 at 12:13 PM, Gary Helmling <[EMAIL PROTECTED]>
> wrote:
> > Something else seems to be going on.  With the call to
> shutdownMiniCluster()
> > the first run of TestHLog passes.  But when I try running in a loop, the
> > second run always seems to hang.
> >
> > Thread dump here: http://pastebin.com/f18Wfa3T
> >
> >
> > On Wed, Oct 5, 2011 at 12:00 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> >
> >> +CC Roman who worked on the patch identified by the bisect.
> >>
> >> Roman, does Gary's analysis make sense to you?
> >>
> >> -Todd
> >>
> >> On Wed, Oct 5, 2011 at 11:55 AM, Gary Helmling <[EMAIL PROTECTED]>
> >> wrote:
> >> > Somehow TestHLog was never actually shutting down the mini-cluster?
> >> >
> >> > The following change lets the test exit successfully:
> >> >
> >> > diff --git
> >> > a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> >> > b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> >> > index 663b318..13f821c 100644
> >> > ---
> >> a/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> >> > +++
> >> b/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> >> > @@ -54,6 +54,7 @@ import
> >> > org.apache.hadoop.hdfs.server.namenode.LeaseManager;
> >> >  import org.apache.hadoop.io.SequenceFile;
> >> >  import org.apache.log4j.Level;
> >> >  import org.junit.After;
> >> > +import org.junit.AfterClass;
> >> >  import org.junit.Before;
> >> >  import org.junit.BeforeClass;
> >> >  import org.junit.Test;
> >> > @@ -120,6 +121,11 @@ public class TestHLog  {
> >> >     oldLogDir = new Path(hbaseDir, ".oldlogs");
> >> >     dir = new Path(hbaseDir, getName());
> >> >   }
> >> > +  @AfterClass
> >> > +  public static void tearDownAfterClass() throws Exception {
> >> > +    TEST_UTIL.shutdownMiniCluster();
> >> > +  }
> >> > +
> >> >   private static String getName() {
> >> >     // TODO Auto-generated method stub
> >> >     return "TestHLog";
>

On Wed, Oct 5, 2011 at 12:00 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> +CC Roman who worked on the patch identified by the bisect.

Thanks for the CC. I've just subscribed myself to the list.

> Roman, does Gary's analysis make sense to you?

Hm. I need to look into it. There *could* be some issues with the mini-cluster,
since I've enabled a whole new codepath for it.

Stay tuned...

Thanks,
Roman.

On Wed, Oct 5, 2011 at 3:47 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:

> On Wed, Oct 5, 2011 at 12:00 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> > +CC Roman who worked on the patch identified by the bisect.
>
> Thanks for the CC. I've just subscribed myself to the list.
>
> > Roman, does Gary's analysis make sense to you?
>
> Hm. I need to look into it. There *could* be some issues with the
> mini-cluster,
> since I've enabled a whole new codepath for it.
>
>
>
I'm thinking this may just be sloppiness or lack of cleanup in TestHLog that
was just exposed by the HBASE-4209 change.  Even if I revert the change to
JVMClusterUtil from 4209 and instead use the patch to call shutdown in
TestHLog.tearDownAfterClass(), I still get hangs every 3-4 runs when
executing TestHLog in a loop.

Previously, when TestHLog had neither the shutdown hook nor an explicit
shutdown of the mini cluster, I'm guessing that the test would "pass" even
though the mini cluster never fully shut down.  So it's good to have the
shutdown hooks in place for tests to help us ferret out these issues.

I created HBASE-4545 to clean up TestHLog.

Among other things, it didn't even need to spin up a MiniHBaseCluster, only
a MiniDFSCluster.  Several of the tests were also leaking unclosed HLog
instances.

Will post a patch shortly.
On Wed, Oct 5, 2011 at 5:52 PM, Gary Helmling <[EMAIL PROTECTED]> wrote:

> On Wed, Oct 5, 2011 at 3:47 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
>
>> On Wed, Oct 5, 2011 at 12:00 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>> > +CC Roman who worked on the patch identified by the bisect.
>>
>> Thanks for the CC. I've just subscribed myself to the list.
>>
>> > Roman, does Gary's analysis make sense to you?
>>
>> Hm. I need to look into it. There *could* be some issues with the
>> mini-cluster,
>> since I've enabled a whole new codepath for it.
>>
>>
>>
> I'm thinking this may just be sloppiness or lack of cleanup in TestHLog
> that was just exposed by the HBASE-4209 change.  Even if I revert the change
> to JVMClusterUtil from 4209 and instead use the patch to call shutdown in
> TestHLog.tearDownAfterClass(), I still get hangs every 3-4 runs when
> executing TestHLog in a loop.
>
> Previously, when TestHLog had neither the shutdown hook nor an explicit
> shutdown of the mini cluster, I'm guessing that the test would "pass" even
> though the mini cluster never fully shut down.  So it's good to have the
> shutdown hooks in place for tests to help us ferret out these issues.
>
>
>

Good stuff G.
St.Ack

On Wed, Oct 5, 2011 at 8:44 PM, Gary Helmling <[EMAIL PROTECTED]> wrote:
> I created HBASE-4545 to clean up TestHLog.
>
> Among other things, it didn't even need to spin up a MiniHBaseCluster, only
> a MiniDFSCluster.  Several of the tests were also leaking unclosed HLog
> instances.
>
> Will post a patch shortly.
>
>
> On Wed, Oct 5, 2011 at 5:52 PM, Gary Helmling <[EMAIL PROTECTED]> wrote:
>
>> On Wed, Oct 5, 2011 at 3:47 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote:
>>
>>> On Wed, Oct 5, 2011 at 12:00 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>> > +CC Roman who worked on the patch identified by the bisect.
>>>
>>> Thanks for the CC. I've just subscribed myself to the list.
>>>
>>> > Roman, does Gary's analysis make sense to you?
>>>
>>> Hm. I need to look into it. There *could* be some issues with the
>>> mini-cluster,
>>> since I've enabled a whole new codepath for it.
>>>
>>>
>>>
>> I'm thinking this may just be sloppiness or lack of cleanup in TestHLog
>> that was just exposed by the HBASE-4209 change.  Even if I revert the change
>> to JVMClusterUtil from 4209 and instead use the patch to call shutdown in
>> TestHLog.tearDownAfterClass(), I still get hangs every 3-4 runs when
>> executing TestHLog in a loop.
>>
>> Previously, when TestHLog had neither the shutdown hook nor an explicit
>> shutdown of the mini cluster, I'm guessing that the test would "pass" even
>> though the mini cluster never fully shut down.  So it's good to have the
>> shutdown hooks in place for tests to help us ferret out these issues.
>>
>>
>>
>

Hi Gary,

On Wed, Oct 5, 2011 at 8:44 PM, Gary Helmling <[EMAIL PROTECTED]> wrote:
> I created HBASE-4545 to clean up TestHLog.
>
> Among other things, it didn't even need to spin up a MiniHBaseCluster, only
> a MiniDFSCluster.  Several of the tests were also leaking unclosed HLog
> instances.
>
> Will post a patch shortly.

Thanks for taking a look at this. I'm still a bit new around those areas of
the code base. But I'm learning ;-)

Thanks,
Roman.