Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # dev - All region server died due to "Parent directory doesn't exist"


+
lars hofhansl 2013-05-09, 06:39
+
lars hofhansl 2013-05-09, 07:23
+
lars hofhansl 2013-05-09, 07:41
+
Ted Yu 2013-05-09, 08:33
+
Andrew Purtell 2013-05-09, 08:59
+
Ted Yu 2013-05-09, 09:04
+
Andrew Purtell 2013-05-09, 09:06
+
lars hofhansl 2013-05-09, 15:48
+
Ted Yu 2013-05-09, 16:07
+
lars hofhansl 2013-05-09, 16:16
+
Varun Sharma 2013-05-09, 16:39
+
Varun Sharma 2013-05-09, 16:41
+
Ted Yu 2013-05-09, 16:51
+
lars hofhansl 2013-05-09, 17:03
+
Stack 2013-05-09, 17:34
+
lars hofhansl 2013-05-09, 18:13
+
lars hofhansl 2013-05-09, 18:28
+
Enis Söztutar 2013-05-10, 01:10
+
lars hofhansl 2013-05-10, 04:25
+
Enis Söztutar 2013-05-10, 05:01
+
lars hofhansl 2013-05-10, 05:47
Copy link to this message
-
Re: All region server died due to "Parent directory doesn't exist"
lars hofhansl 2013-05-09, 16:38
Another symptom is that about 1h before the RSs started dying I get logs like this:
2013-05-08 15:02:50,723 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Unable to open a reader, sleeping 1000 times
Replication is not the problem here, but it indicates that it suddenly cannot no longer read the log files.
There is nothing interesting in the master log, and as I said HDFS is fine.

-- Lars

----- Original Message -----
From: lars hofhansl <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Cc:
Sent: Thursday, May 9, 2013 9:16 AM
Subject: Re: All region server died due to "Parent directory doesn't exist"

Thanks Ted. I'll do the same.
----- Original Message -----
From: Ted Yu <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
Cc:
Sent: Thursday, May 9, 2013 9:07 AM
Subject: Re: All region server died due to "Parent directory doesn't exist"

I went through the patch for HBASE-7824 one more time and didn't find
direct correlation to the issue Lars reported.

I am going over the other JIRAs in Lars' list.

Cheers

On Thu, May 9, 2013 at 8:48 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> I will try. I do not think this is the issue, though.
>
> The master is up in my case.
> Right now the cluster is in a state where each region server aborts itself
> shortly after being started (which coincides with having it's log directory
> renamed to ...-splitting).
>
>
> This is a test cluster and I could just start from scratch... This appears
> to be a serious enough problem, though, and I would like to track down the
> issue.
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Ted Yu <[EMAIL PROTECTED]>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Sent: Thursday, May 9, 2013 2:04 AM
> Subject: Re: All region server died due to "Parent directory doesn't exist"
>
> The config came from hbase-7824.
>
> There are other JIRAs in Lars' list which are related to log splitting.
>
> I think more investigation is needed.
>
> Cheers
>
> On May 9, 2013, at 1:59 AM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
>
> > So that is HBASE-7824, right?
> >
> > On Thu, May 9, 2013 at 4:33 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> >> hbase.master.wait.for.log.splitting
> >
> >
> >
> >
> > --
> > Best regards,
> >
> >   - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>
>
+
takeshi 2014-02-19, 03:18