Hello,

I created an Hbase table programatically like so:

String tableName = "_myTable"
HBaseAdmin admin = new HBaseAdmin(<some_configuration>);
if (admin.tableExists(outTable) == false) {
HTableDescriptor desc = new HTableDescriptor(tableName);
HColumnDescriptor count = new HColumnDescriptor(Bytes.toBytes("t"));
desc.addFamily(count);
admin.createTable(desc);
}

However, when I use the table and also in the HBase logs I find warnings
like this at regular intervals:

WARN client.HTable: Null regioninfo cell in keyvalues={_myTable
,,1344874729874.61499712e41143df595f2215d1ab7b91./info:server/1344876299527/Put/vlen=29,
_myTable
,,1344874729874.61499712e41143df595f2215d1ab7b91./info:serverstartcode/1344876299527/Put/vlen=8}

Is there something wrong with the way I created the table?

Thank you,

/David

Hey David,

Are you running your HBase instance in
pseudo-distributed/fully-distributed mode or the standalone one?

Does "hbase hbck" report any inconsistencies?

On Tue, Aug 14, 2012 at 7:40 PM, David Koch <[EMAIL PROTECTED]> wrote:
> Hello,
>
> I created an Hbase table programatically like so:
>
> String tableName = "_myTable"
> HBaseAdmin admin = new HBaseAdmin(<some_configuration>);
> if (admin.tableExists(outTable) == false) {
> HTableDescriptor desc = new HTableDescriptor(tableName);
> HColumnDescriptor count = new HColumnDescriptor(Bytes.toBytes("t"));
> desc.addFamily(count);
> admin.createTable(desc);
> }
>
> However, when I use the table and also in the HBase logs I find warnings
> like this at regular intervals:
>
> WARN client.HTable: Null regioninfo cell in keyvalues={_myTable
> ,,1344874729874.61499712e41143df595f2215d1ab7b91./info:server/1344876299527/Put/vlen=29,
> _myTable
> ,,1344874729874.61499712e41143df595f2215d1ab7b91./info:serverstartcode/1344876299527/Put/vlen=8}
>
> Is there something wrong with the way I created the table?
>
> Thank you,
>
> /David

--
Harsh J

Hello,

It's a fully-distributed environment (CDH3). Hbase hbck sometimes reports
inconsistencies like:

ERROR: Region { meta =>
_myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.,
hdfs => null, deployed =>
stag-back-9.ezakus.net:60020;_myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.
} found in META, but not in HDFS, and deployed on
<server_name_replaced>:60020

However, in the end it always tells me the table is ok. Only for _myTable
does the master log contain the warning messages mentioned in my original
post, one every 5 minutes:

2012-08-14 23:53:53,167 WARN org.apache.hadoop.hbase.master.CatalogJanitor:
REGIONINFO_QUALIFIER is empty in keyvalues={_myTable
,,1344874729874.61499712e41143df595f2215d1ab7b91./info:server/1344876299527/Put/vlen=29,
_myTable
,,1344874729874.61499712e41143df595f2215d1ab7b91./info:serverstartcode/1344876299527/Put/vlen=8}

The region server log where _myTable is hosted contains the following:

2012-08-14 11:36:31,801 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to
open region: _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.
2012-08-14 11:36:31,832 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Onlined _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.; next
sequenceid=1
2012-08-14 11:36:31,860 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
Updated row _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27. in
region .META.,,1 with server=<server_name_replaced>:60020,
startcode=1344877239935

I remember disabling the table from base shell with the plan to drop it
later - I may then have launched the application which (re-)creates
_myTable if it does not exist without ever having issued the drop before.
Not sure if that could be part of the issue.

The data can be re-generated quickly. If it gets rid of the warning I can
just disable and drop the table. Would I need to carry out any other
maintenance tasks like editing .META before re-creating a "clean" version
of this table?

Thanks,

/David

On Tue, Aug 14, 2012 at 5:48 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Hey David,
>
> Are you running your HBase instance in
> pseudo-distributed/fully-distributed mode or the standalone one?
>
> Does "hbase hbck" report any inconsistencies?
>
> On Tue, Aug 14, 2012 at 7:40 PM, David Koch <[EMAIL PROTECTED]> wrote:
> > Hello,
> >
> > I created an Hbase table programatically like so:
> >
> > String tableName = "_myTable"
> > HBaseAdmin admin = new HBaseAdmin(<some_configuration>);
> > if (admin.tableExists(outTable) == false) {
> > HTableDescriptor desc = new HTableDescriptor(tableName);
> > HColumnDescriptor count = new HColumnDescriptor(Bytes.toBytes("t"));
> > desc.addFamily(count);
> > admin.createTable(desc);
> > }
> >
> > However, when I use the table and also in the HBase logs I find warnings
> > like this at regular intervals:
> >
> > WARN client.HTable: Null regioninfo cell in keyvalues={_myTable
> >
> ,,1344874729874.61499712e41143df595f2215d1ab7b91./info:server/1344876299527/Put/vlen=29,
> > _myTable
> >
> ,,1344874729874.61499712e41143df595f2215d1ab7b91./info:serverstartcode/1344876299527/Put/vlen=8}
> >
> > Is there something wrong with the way I created the table?
> >
> > Thank you,
> >
> > /David
>
>
>
> --
> Harsh J

On Tue, Aug 14, 2012 at 7:10 AM, David Koch <[EMAIL PROTECTED]> wrote:
> Hello,
>
> I created an Hbase table programatically like so:
>
> String tableName = "_myTable"
> HBaseAdmin admin = new HBaseAdmin(<some_configuration>);
> if (admin.tableExists(outTable) == false) {
> HTableDescriptor desc = new HTableDescriptor(tableName);
> HColumnDescriptor count = new HColumnDescriptor(Bytes.toBytes("t"));
> desc.addFamily(count);
> admin.createTable(desc);
> }
>
> However, when I use the table and also in the HBase logs I find warnings
> like this at regular intervals:
>
> WARN client.HTable: Null regioninfo cell in keyvalues={_myTable
> ,,1344874729874.61499712e41143df595f2215d1ab7b91./info:server/1344876299527/Put/vlen=29,
> _myTable
> ,,1344874729874.61499712e41143df595f2215d1ab7b91./info:serverstartcode/1344876299527/Put/vlen=8}
>
> Is there something wrong with the way I created the table?
>

No.

Seems like a row in .META. has not been properly cleared.  Any
error/problem in master logs before the first instance of the above
log?

St.Ack

On Tue, Aug 14, 2012 at 3:42 PM, David Koch <[EMAIL PROTECTED]> wrote:
> Hello,
>
> It's a fully-distributed environment (CDH3). Hbase hbck sometimes reports
> inconsistencies like:
>
> ERROR: Region { meta =>
> _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.,
> hdfs => null, deployed =>
> stag-back-9.ezakus.net:60020;_myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.
> } found in META, but not in HDFS, and deployed on
> <server_name_replaced>:60020
>
> However, in the end it always tells me the table is ok.

During normal operation, transitory conditions can look like problems to hbck.

> Only for _myTable
> does the master log contain the warning messages mentioned in my original
> post, one every 5 minutes:
>

The CatalogJanitor runs every 5 minutes.  Its the thing that does
checks and janitorial tasks on the .META. table.
> 2012-08-14 23:53:53,167 WARN org.apache.hadoop.hbase.master.CatalogJanitor:
> REGIONINFO_QUALIFIER is empty in keyvalues={_myTable
> ,,1344874729874.61499712e41143df595f2215d1ab7b91./info:server/1344876299527/Put/vlen=29,
> _myTable
> ,,1344874729874.61499712e41143df595f2215d1ab7b91./info:serverstartcode/1344876299527/Put/vlen=8}
>
> The region server log where _myTable is hosted contains the following:
>
> 2012-08-14 11:36:31,801 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to
> open region: _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.
> 2012-08-14 11:36:31,832 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Onlined _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.; next
> sequenceid=1
> 2012-08-14 11:36:31,860 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
> Updated row _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27. in
> region .META.,,1 with server=<server_name_replaced>:60020,
> startcode=1344877239935
>
> I remember disabling the table from base shell with the plan to drop it
> later - I may then have launched the application which (re-)creates
> _myTable if it does not exist without ever having issued the drop before.
> Not sure if that could be part of the issue.
>

Yes.  Could be.  You could delete the rows in .META. that mention your
__myTable (Be careful you only delete this row).  That should fix the
master log message.

> The data can be re-generated quickly. If it gets rid of the warning I can
> just disable and drop the table. Would I need to carry out any other
> maintenance tasks like editing .META before re-creating a "clean" version
> of this table?
>

Just do the above delete.  That should take care of it.

Find the row name in .META. that mentions your __myTable (is it
"_myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27."?)

For example:

hbase> deleteall ".META.",
"_myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27."

St.Ack

Hello,

Thank you for your detailed response.

I did the delete in .META. - the table does now not exist anymore according
to hbase hbck and hbase shell however the warning message persists.

/David

On Wed, Aug 15, 2012 at 11:16 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Tue, Aug 14, 2012 at 3:42 PM, David Koch <[EMAIL PROTECTED]> wrote:
> > Hello,
> >
> > It's a fully-distributed environment (CDH3). Hbase hbck sometimes reports
> > inconsistencies like:
> >
> > ERROR: Region { meta =>
> > _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.,
> > hdfs => null, deployed =>
> > stag-back-9.ezakus.net:60020
> ;_myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.
> > } found in META, but not in HDFS, and deployed on
> > <server_name_replaced>:60020
> >
> > However, in the end it always tells me the table is ok.
>
> During normal operation, transitory conditions can look like problems to
> hbck.
>
> > Only for _myTable
> > does the master log contain the warning messages mentioned in my original
> > post, one every 5 minutes:
> >
>
> The CatalogJanitor runs every 5 minutes.  Its the thing that does
> checks and janitorial tasks on the .META. table.
>
>
> > 2012-08-14 23:53:53,167 WARN
> org.apache.hadoop.hbase.master.CatalogJanitor:
> > REGIONINFO_QUALIFIER is empty in keyvalues={_myTable
> >
> ,,1344874729874.61499712e41143df595f2215d1ab7b91./info:server/1344876299527/Put/vlen=29,
> > _myTable
> >
> ,,1344874729874.61499712e41143df595f2215d1ab7b91./info:serverstartcode/1344876299527/Put/vlen=8}
> >
> > The region server log where _myTable is hosted contains the following:
> >
> > 2012-08-14 11:36:31,801 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to
> > open region: _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.
> > 2012-08-14 11:36:31,832 INFO
> org.apache.hadoop.hbase.regionserver.HRegion:
> > Onlined _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27.; next
> > sequenceid=1
> > 2012-08-14 11:36:31,860 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
> > Updated row _myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27. in
> > region .META.,,1 with server=<server_name_replaced>:60020,
> > startcode=1344877239935
> >
> > I remember disabling the table from base shell with the plan to drop it
> > later - I may then have launched the application which (re-)creates
> > _myTable if it does not exist without ever having issued the drop before.
> > Not sure if that could be part of the issue.
> >
>
> Yes.  Could be.  You could delete the rows in .META. that mention your
> __myTable (Be careful you only delete this row).  That should fix the
> master log message.
>
> > The data can be re-generated quickly. If it gets rid of the warning I can
> > just disable and drop the table. Would I need to carry out any other
> > maintenance tasks like editing .META before re-creating a "clean" version
> > of this table?
> >
>
> Just do the above delete.  That should take care of it.
>
> Find the row name in .META. that mentions your __myTable (is it
> "_myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27."?)
>
> For example:
>
> hbase> deleteall ".META.",
> "_myTable,,1344936991240.979b3fe3ced9016372a82b7af5d33c27."
>
> St.Ack
>

On Thu, Aug 16, 2012 at 5:46 AM, David Koch <[EMAIL PROTECTED]> wrote:
> Hello,
>
> Thank you for your detailed response.
>
> I did the delete in .META. - the table does now not exist anymore according
> to hbase hbck and hbase shell however the warning message persists.
>

Perhaps another row still in .META.?  If you scan .META., do you see one?

hbase> scan '.META.'

St.Ack

You were right, there was another row in .META. that needed to be deleted.
The warning messages have stopped. Thank you.

/David

On Thu, Aug 16, 2012 at 5:47 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Thu, Aug 16, 2012 at 5:46 AM, David Koch <[EMAIL PROTECTED]> wrote:
> > Hello,
> >
> > Thank you for your detailed response.
> >
> > I did the delete in .META. - the table does now not exist anymore
> according
> > to hbase hbck and hbase shell however the warning message persists.
> >
>
> Perhaps another row still in .META.?  If you scan .META., do you see one?
>
> hbase> scan '.META.'
>
> St.Ack
>