Hi,

When I execute command of 'list' in hbase shell, it told me that the HRegioninfo was null or empty in -ROOT-.
Did the -ROOT- table value got corrupted or any other reasons cause this problem?

Here's the info from the 'list' and 'hbck' tool:
--------------------------------------------------------------------------------------------------------------------
hbase(main):001:0> list
TABLE
11/04/29 10:46:23 INFO jute.BinaryInputArchive: Expecting maxBufferString as an integer but got as null.Hence setting to default value 1048575

ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 7 attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
--------------------------------------------------------------------------------------------------------------------
HBase hbck tool:
--------------------------------------------------------------------------------------------------------------------
11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=4 of 10 failed; retrying after sleep of 2000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
11/04/29 10:45:42 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=5 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
11/04/29 10:45:44 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=6 of 10 failed; retrying after sleep of 4000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
11/04/29 10:45:48 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10 failed; retrying after sleep of 8000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vlen=13, .META.,,1/info:serverstartcode/1304015789456/Put/vlen=8}
11/04/29 10:45:52 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@42787d6a; hsa=Linux-1:60020
11/04/29 10:46:00 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation=address: Linux-1:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10 failed; retrying after sleep of 16000 because: HRegionInfo was null or empty in -ROOT-, row=keyvalues={.META.,,1/info:server/1304015789456/Put/vl

Hi all,

This morning I was restarting my HBase cluster, and I'm finding that the
region servers start up fine, but master always fails with the following
error shown in the web ui:

Problem accessing /master.jsp. Reason:

    Trying to contact region server null for region , row '', but
failed after 10 attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in -ROOT-,
row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
.META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-,
row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
.META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
...

Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
contact region server null for region , row '', but failed after 10
attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in -ROOT-,
row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
.META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
java.io.IOException: HRegionInfo was null or empty in -ROOT-,
row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
.META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
...

Here is what I get when I scan -ROOT-:
hbase(main):001:0> scan '-ROOT-'
ROW                                     COLUMN+CELL
 .META.,,1                              column=info:server,
timestamp=1301428760725, value=sspv2rndhadoop1.gce2000.com:60020
 .META.,,1                              column=info:serverstartcode,
timestamp=1301428760725, value=1301428748891

Any ideas what's going on / how I can fix this?
Cluster info: 4 nodes, hadoop-0.20-append, hbase-0.90.3, red hat 5.5,
64-bit, jdk1.6; 64 gb ram, 2.8ghz.
Thanks guys

Can you check master log and see whats happening around -ROOT-? Why is
it not being assigned out?  What if you kill master process and
restart it?  Does -ROOT- get assigned then?

St.Ack

On Wed, Mar 30, 2011 at 7:21 AM, Nichole Treadway <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> This morning I was restarting my HBase cluster, and I'm finding that the
> region servers start up fine, but master always fails with the following
> error shown in the web ui:
>
> Problem accessing /master.jsp. Reason:
>
>    Trying to contact region server null for region , row '', but
> failed after 10 attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in -ROOT-,
> row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
> .META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-,
> row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
> .META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
> ...
>
> Caused by:
>
>
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> contact region server null for region , row '', but failed after 10
> attempts.
> Exceptions:
> java.io.IOException: HRegionInfo was null or empty in -ROOT-,
> row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
> .META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
> java.io.IOException: HRegionInfo was null or empty in -ROOT-,
> row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
> .META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
>
>
> ...
>
> Here is what I get when I scan -ROOT-:
>
>
> hbase(main):001:0> scan '-ROOT-'
> ROW                                     COLUMN+CELL
>  .META.,,1                              column=info:server,
> timestamp=1301428760725, value=sspv2rndhadoop1.gce2000.com:60020
>  .META.,,1                              column=info:serverstartcode,
> timestamp=1301428760725, value=1301428748891
>
>
>
> Any ideas what's going on / how I can fix this?
>
>
> Cluster info: 4 nodes, hadoop-0.20-append, hbase-0.90.3, red hat 5.5,
> 64-bit, jdk1.6; 64 gb ram, 2.8ghz.
>
>
> Thanks guys
>

In the master logs, I see messages like I've pasted below. I get the same
behavior even when I stop master and restart it. I see the root and meta
table regions listed in the region server ui's

2011-03-30 13:15:00,675 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation=address:
node1.abc123.com:60020, regioninfo: -ROOT-,,0.70236052, attempt=7 of 10
failed; retrying after sleep of 8000 because: HRegionInfo was null or empty
in -ROOT-, row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
.META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
2011-03-30 13:15:00,675 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Lookedup root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@475a07bf;
hsa=node1.abc123.com:60020
2011-03-30 13:15:08,679 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Lookedup root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@475a07bf;
hsa=node1.abc123.com:60020
2011-03-30 13:15:08,681 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation=address:
node1.abc123.com:60020, regioninfo: -ROOT-,,0.70236052, attempt=8 of 10
failed; retrying after sleep of 16000 because: HRegionInfo was null or empty
in -ROOT-, row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
.META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
2011-03-30 13:15:08,681 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Lookedup root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@475a07bf;
hsa=node1.abc123.com:60020
2011-03-30 13:15:24,685 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Lookedup root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@475a07bf;
hsa=node1.abc123.com:60020
2011-03-30 13:15:24,688 WARN org.mortbay.log: /master.jsp:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server null for region , row '', but failed after 10 attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in -ROOT-,
row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
.META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
...
2011-03-30 13:19:32,344 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-03-30 13:19:32,346 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing.  servers=4 regions=2 average=0.5 mostloaded=1
leastloaded=1

On Wed, Mar 30, 2011 at 12:19 PM, Stack <[EMAIL PROTECTED]> wrote:

> Can you check master log and see whats happening around -ROOT-? Why is
> it not being assigned out?  What if you kill master process and
> restart it?  Does -ROOT- get assigned then?
>
> St.Ack
>
> On Wed, Mar 30, 2011 at 7:21 AM, Nichole Treadway <[EMAIL PROTECTED]>
> wrote:
> > Hi all,
> >
> > This morning I was restarting my HBase cluster, and I'm finding that the
> > region servers start up fine, but master always fails with the following
> > error shown in the web ui:
> >
> > Problem accessing /master.jsp. Reason:
> >
> >    Trying to contact region server null for region , row '', but
> > failed after 10 attempts.
> > Exceptions:
> > java.io.IOException: HRegionInfo was null or empty in -ROOT-,
> > row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
> > .META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
> > java.io.IOException: HRegionInfo was null or empty in -ROOT-,
> > row=keyvalues={.META.,,1/info:server/1301428760725/Put/vlen=33,
> > .META.,,1/info:serverstartcode/1301428760725/Put/vlen=8}
> > ...
> >
> > Caused by:
> >
> >
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to