Hi,

I'm wondering, how much should we worrie about those warnings:
WARN org.apache.hadoop.hbase.master.CatalogJanitor: REGIONINFO_QUALIFIER is
empty in keyvalues={

I have 6 of them show each time the CatalgoJanitor is running (5 minutes).

Doesn't seems to really be an issue, but then why is it a WARN?

Thanks,

JM

We recently saw some of these warnings in a cluster we were setting up.
These warnings mean there are rows in the META table that are missing one
of the expected columns.  In our case, we verified that these regions
didn't appear to exist in HDFS either and the table itself showed no holes
or problems in hbck or via our application.  After looking at the
timestamps of the other data in these rows in META we determined that the
remaining columns in META had been written on an earlier incarnation of a
table with the same name.  I.e. we created a table once, put data in it,
deleted the table, created a new table of the same name, and apparently not
everything was correctly deleted from META.  I'm not sure how that
happened, but to clean it up we manually removed those rows from META and
didn't observe any more WARNings or problems afterward.  There's another
mailing list thread from some months back from someone with a similar issue
and resolution.  Good luck!

Dave
On Sun, Feb 24, 2013 at 5:03 PM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I'm wondering, how much should we worrie about those warnings:
> WARN org.apache.hadoop.hbase.master.CatalogJanitor: REGIONINFO_QUALIFIER is
> empty in keyvalues={
>
> I have 6 of them show each time the CatalgoJanitor is running (5 minutes).
>
> Doesn't seems to really be an issue, but then why is it a WARN?
>
> Thanks,
>
> JM
>

Thanks for the info Dave.  Others will be benefited too.

Regards
Ram

On Mon, Feb 25, 2013 at 7:25 PM, Dave Latham <[EMAIL PROTECTED]> wrote:

> We recently saw some of these warnings in a cluster we were setting up.
> These warnings mean there are rows in the META table that are missing one
> of the expected columns.  In our case, we verified that these regions
> didn't appear to exist in HDFS either and the table itself showed no holes
> or problems in hbck or via our application.  After looking at the
> timestamps of the other data in these rows in META we determined that the
> remaining columns in META had been written on an earlier incarnation of a
> table with the same name.  I.e. we created a table once, put data in it,
> deleted the table, created a new table of the same name, and apparently not
> everything was correctly deleted from META.  I'm not sure how that
> happened, but to clean it up we manually removed those rows from META and
> didn't observe any more WARNings or problems afterward.  There's another
> mailing list thread from some months back from someone with a similar issue
> and resolution.  Good luck!
>
> Dave
>
>
> On Sun, Feb 24, 2013 at 5:03 PM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > I'm wondering, how much should we worrie about those warnings:
> > WARN org.apache.hadoop.hbase.master.CatalogJanitor: REGIONINFO_QUALIFIER
> is
> > empty in keyvalues={
> >
> > I have 6 of them show each time the CatalgoJanitor is running (5
> minutes).
> >
> > Doesn't seems to really be an issue, but then why is it a WARN?
> >
> > Thanks,
> >
> > JM
> >
>

Hi Dave,

Thanks for the details.

Is it something that we should think about adding into HBCK?

Seems that it can detect them "Number of empty REGIONINFO_QUALIFIER rows in
.META.: 6" but it's not fixing them nor listing them.

JM

2013/2/25 ramkrishna vasudevan <[EMAIL PROTECTED]>

> Thanks for the info Dave.  Others will be benefited too.
>
> Regards
> Ram
>
> On Mon, Feb 25, 2013 at 7:25 PM, Dave Latham <[EMAIL PROTECTED]> wrote:
>
> > We recently saw some of these warnings in a cluster we were setting up.
> > These warnings mean there are rows in the META table that are missing one
> > of the expected columns.  In our case, we verified that these regions
> > didn't appear to exist in HDFS either and the table itself showed no
> holes
> > or problems in hbck or via our application.  After looking at the
> > timestamps of the other data in these rows in META we determined that the
> > remaining columns in META had been written on an earlier incarnation of a
> > table with the same name.  I.e. we created a table once, put data in it,
> > deleted the table, created a new table of the same name, and apparently
> not
> > everything was correctly deleted from META.  I'm not sure how that
> > happened, but to clean it up we manually removed those rows from META and
> > didn't observe any more WARNings or problems afterward.  There's another
> > mailing list thread from some months back from someone with a similar
> issue
> > and resolution.  Good luck!
> >
> > Dave
> >
> >
> > On Sun, Feb 24, 2013 at 5:03 PM, Jean-Marc Spaggiari <
> > [EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > I'm wondering, how much should we worrie about those warnings:
> > > WARN org.apache.hadoop.hbase.master.CatalogJanitor:
> REGIONINFO_QUALIFIER
> > is
> > > empty in keyvalues={
> > >
> > > I have 6 of them show each time the CatalgoJanitor is running (5
> > minutes).
> > >
> > > Doesn't seems to really be an issue, but then why is it a WARN?
> > >
> > > Thanks,
> > >
> > > JM
> > >
> >
>

I think hbck should be enhanced to deal with this situation.

On Mon, Feb 25, 2013 at 6:56 AM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> Hi Dave,
>
> Thanks for the details.
>
> Is it something that we should think about adding into HBCK?
>
> Seems that it can detect them "Number of empty REGIONINFO_QUALIFIER rows in
> .META.: 6" but it's not fixing them nor listing them.
>
> JM
>
> 2013/2/25 ramkrishna vasudevan <[EMAIL PROTECTED]>
>
> > Thanks for the info Dave.  Others will be benefited too.
> >
> > Regards
> > Ram
> >
> > On Mon, Feb 25, 2013 at 7:25 PM, Dave Latham <[EMAIL PROTECTED]>
> wrote:
> >
> > > We recently saw some of these warnings in a cluster we were setting up.
> > > These warnings mean there are rows in the META table that are missing
> one
> > > of the expected columns.  In our case, we verified that these regions
> > > didn't appear to exist in HDFS either and the table itself showed no
> > holes
> > > or problems in hbck or via our application.  After looking at the
> > > timestamps of the other data in these rows in META we determined that
> the
> > > remaining columns in META had been written on an earlier incarnation
> of a
> > > table with the same name.  I.e. we created a table once, put data in
> it,
> > > deleted the table, created a new table of the same name, and apparently
> > not
> > > everything was correctly deleted from META.  I'm not sure how that
> > > happened, but to clean it up we manually removed those rows from META
> and
> > > didn't observe any more WARNings or problems afterward.  There's
> another
> > > mailing list thread from some months back from someone with a similar
> > issue
> > > and resolution.  Good luck!
> > >
> > > Dave
> > >
> > >
> > > On Sun, Feb 24, 2013 at 5:03 PM, Jean-Marc Spaggiari <
> > > [EMAIL PROTECTED]> wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm wondering, how much should we worrie about those warnings:
> > > > WARN org.apache.hadoop.hbase.master.CatalogJanitor:
> > REGIONINFO_QUALIFIER
> > > is
> > > > empty in keyvalues={
> > > >
> > > > I have 6 of them show each time the CatalgoJanitor is running (5
> > > minutes).
> > > >
> > > > Doesn't seems to really be an issue, but then why is it a WARN?
> > > >
> > > > Thanks,
> > > >
> > > > JM
> > > >
> > >
> >
>

Ok. Document a JIRA and take a look at hbck to add that.

JM

2013/2/25 Ted Yu <[EMAIL PROTECTED]>

> I think hbck should be enhanced to deal with this situation.
>
> On Mon, Feb 25, 2013 at 6:56 AM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
> > Hi Dave,
> >
> > Thanks for the details.
> >
> > Is it something that we should think about adding into HBCK?
> >
> > Seems that it can detect them "Number of empty REGIONINFO_QUALIFIER rows
> in
> > .META.: 6" but it's not fixing them nor listing them.
> >
> > JM
> >
> > 2013/2/25 ramkrishna vasudevan <[EMAIL PROTECTED]>
> >
> > > Thanks for the info Dave.  Others will be benefited too.
> > >
> > > Regards
> > > Ram
> > >
> > > On Mon, Feb 25, 2013 at 7:25 PM, Dave Latham <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > We recently saw some of these warnings in a cluster we were setting
> up.
> > > > These warnings mean there are rows in the META table that are missing
> > one
> > > > of the expected columns.  In our case, we verified that these regions
> > > > didn't appear to exist in HDFS either and the table itself showed no
> > > holes
> > > > or problems in hbck or via our application.  After looking at the
> > > > timestamps of the other data in these rows in META we determined that
> > the
> > > > remaining columns in META had been written on an earlier incarnation
> > of a
> > > > table with the same name.  I.e. we created a table once, put data in
> > it,
> > > > deleted the table, created a new table of the same name, and
> apparently
> > > not
> > > > everything was correctly deleted from META.  I'm not sure how that
> > > > happened, but to clean it up we manually removed those rows from META
> > and
> > > > didn't observe any more WARNings or problems afterward.  There's
> > another
> > > > mailing list thread from some months back from someone with a similar
> > > issue
> > > > and resolution.  Good luck!
> > > >
> > > > Dave
> > > >
> > > >
> > > > On Sun, Feb 24, 2013 at 5:03 PM, Jean-Marc Spaggiari <
> > > > [EMAIL PROTECTED]> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'm wondering, how much should we worrie about those warnings:
> > > > > WARN org.apache.hadoop.hbase.master.CatalogJanitor:
> > > REGIONINFO_QUALIFIER
> > > > is
> > > > > empty in keyvalues={
> > > > >
> > > > > I have 6 of them show each time the CatalgoJanitor is running (5
> > > > minutes).
> > > > >
> > > > > Doesn't seems to really be an issue, but then why is it a WARN?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > JM
> > > > >
> > > >
> > >
> >
>

I have created HBASE-7930 for that.

I have done the patch for 0.94 and build is in progress. I will test
it and do the patch for trunk if tests are successful.

JM

2013/2/25 Jean-Marc Spaggiari <[EMAIL PROTECTED]>
>
> Ok. Document a JIRA and take a look at hbck to add that.
>
> JM
>
>
> 2013/2/25 Ted Yu <[EMAIL PROTECTED]>
>>
>> I think hbck should be enhanced to deal with this situation.
>>
>> On Mon, Feb 25, 2013 at 6:56 AM, Jean-Marc Spaggiari <
>> [EMAIL PROTECTED]> wrote:
>>
>> > Hi Dave,
>> >
>> > Thanks for the details.
>> >
>> > Is it something that we should think about adding into HBCK?
>> >
>> > Seems that it can detect them "Number of empty REGIONINFO_QUALIFIER rows in
>> > .META.: 6" but it's not fixing them nor listing them.
>> >
>> > JM
>> >
>> > 2013/2/25 ramkrishna vasudevan <[EMAIL PROTECTED]>
>> >
>> > > Thanks for the info Dave.  Others will be benefited too.
>> > >
>> > > Regards
>> > > Ram
>> > >
>> > > On Mon, Feb 25, 2013 at 7:25 PM, Dave Latham <[EMAIL PROTECTED]>
>> > wrote:
>> > >
>> > > > We recently saw some of these warnings in a cluster we were setting up.
>> > > > These warnings mean there are rows in the META table that are missing
>> > one
>> > > > of the expected columns.  In our case, we verified that these regions
>> > > > didn't appear to exist in HDFS either and the table itself showed no
>> > > holes
>> > > > or problems in hbck or via our application.  After looking at the
>> > > > timestamps of the other data in these rows in META we determined that
>> > the
>> > > > remaining columns in META had been written on an earlier incarnation
>> > of a
>> > > > table with the same name.  I.e. we created a table once, put data in
>> > it,
>> > > > deleted the table, created a new table of the same name, and apparently
>> > > not
>> > > > everything was correctly deleted from META.  I'm not sure how that
>> > > > happened, but to clean it up we manually removed those rows from META
>> > and
>> > > > didn't observe any more WARNings or problems afterward.  There's
>> > another
>> > > > mailing list thread from some months back from someone with a similar
>> > > issue
>> > > > and resolution.  Good luck!
>> > > >
>> > > > Dave
>> > > >
>> > > >
>> > > > On Sun, Feb 24, 2013 at 5:03 PM, Jean-Marc Spaggiari <
>> > > > [EMAIL PROTECTED]> wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > I'm wondering, how much should we worrie about those warnings:
>> > > > > WARN org.apache.hadoop.hbase.master.CatalogJanitor:
>> > > REGIONINFO_QUALIFIER
>> > > > is
>> > > > > empty in keyvalues={
>> > > > >
>> > > > > I have 6 of them show each time the CatalgoJanitor is running (5
>> > > > minutes).
>> > > > >
>> > > > > Doesn't seems to really be an issue, but then why is it a WARN?
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > JM
>> > > > >
>> > > >
>> > >
>> >
>
>