Hi,

I have a cluster with 10 RS and 1 Master.
From some reason, any call to the HBase fails. I ran hbck and got the same failure:

Tue Nov 06 13:27:07 UTC 2012, org.apache.hadoop.hbase.client.ScannerCallable@7d8a8ce2, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1

at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:196)
at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:77)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:61)
at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntries(HBaseFsck.java:2234)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:334)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
Anyone has any idea how to fix it without the obvious restart?

After restart, running hbck results in:
12/11/07 06:07:28 DEBUG client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@51b4a74b

12/11/07 06:07:28 INFO util.HBaseFsck: getHTableDescriptors == tableNames => []
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.hbase.util.HBaseFsck.reportTablesInFlux(HBaseFsck.java:560)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:346)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)

HBase is version 0.94.0
On Nov 7, 2012, at 7:28 AM, Mesika, Asaf wrote:

> Hi,
>
> I have a cluster with 10 RS and 1 Master.
> From some reason, any call to the HBase fails. I ran hbck and got the same failure:
>
> Tue Nov 06 13:27:07 UTC 2012, org.apache.hadoop.hbase.client.ScannerCallable@7d8a8ce2, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
>
> at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:196)
> at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
> at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
> at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
> at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:77)
> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:61)
> at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntries(HBaseFsck.java:2234)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:334)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
>
>
> Anyone has any idea how to fix it without the obvious restart?
>

The first to try is

hbase hbck -repair

Check here:
http://hbase.apache.org/book/apbs04.html#d2067e11173

Thank you!

Sincerely,
Leonid Fedotov
On Nov 6, 2012, at 10:08 PM, Mesika, Asaf wrote:

> After restart, running hbck results in:
> 12/11/07 06:07:28 DEBUG client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@51b4a74b
>
> 12/11/07 06:07:28 INFO util.HBaseFsck: getHTableDescriptors == tableNames => []
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck.reportTablesInFlux(HBaseFsck.java:560)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:346)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
>
> HBase is version 0.94.0
>
>
> On Nov 7, 2012, at 7:28 AM, Mesika, Asaf wrote:
>
>> Hi,
>>
>> I have a cluster with 10 RS and 1 Master.
>> From some reason, any call to the HBase fails. I ran hbck and got the same failure:
>>
>> Tue Nov 06 13:27:07 UTC 2012, org.apache.hadoop.hbase.client.ScannerCallable@7d8a8ce2, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
>>
>> at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:196)
>> at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>> at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>> at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>> at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:77)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:61)
>> at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntries(HBaseFsck.java:2234)
>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:334)
>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
>> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
>>
>>
>> Anyone has any idea how to fix it without the obvious restart?
>>
>

See my 2nd mail regarding this: Didn't help.
We're still heavily investigating this, so far without any progress.

On Nov 7, 2012, at 5:43 PM, Leonid Fedotov wrote:

> The first to try is
>
> hbase hbck -repair
>
> Check here:
> http://hbase.apache.org/book/apbs04.html#d2067e11173
>
> Thank you!
>
> Sincerely,
> Leonid Fedotov
>
>
> On Nov 6, 2012, at 10:08 PM, Mesika, Asaf wrote:
>
>> After restart, running hbck results in:
>> 12/11/07 06:07:28 DEBUG client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@51b4a74b
>>
>> 12/11/07 06:07:28 INFO util.HBaseFsck: getHTableDescriptors == tableNames => []
>> Exception in thread "main" java.lang.NullPointerException
>> at org.apache.hadoop.hbase.util.HBaseFsck.reportTablesInFlux(HBaseFsck.java:560)
>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:346)
>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
>> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
>>
>> HBase is version 0.94.0
>>
>>
>> On Nov 7, 2012, at 7:28 AM, Mesika, Asaf wrote:
>>
>>> Hi,
>>>
>>> I have a cluster with 10 RS and 1 Master.
>>> From some reason, any call to the HBase fails. I ran hbck and got the same failure:
>>>
>>> Tue Nov 06 13:27:07 UTC 2012, org.apache.hadoop.hbase.client.ScannerCallable@7d8a8ce2, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
>>>
>>> at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
>>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:196)
>>> at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>>> at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>>> at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>>> at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:77)
>>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:61)
>>> at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntries(HBaseFsck.java:2234)
>>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:334)
>>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
>>> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
>>>
>>>
>>> Anyone has any idea how to fix it without the obvious restart?
>>>
>>
>

More updates in case it can help:

In the UI I get the following regions mapping for .META.:
Table Regions

Name Region Server Start Key End Key Requests
.META.,,1.1028785192 172.25.98.66:11401 -
When I run 'hbase hbck' I fail, with the following error which indicates is thinks this region is placed in a different server (65 and not 66)

12/11/08 14:23:02 DEBUG client.HConnectionManager$HConnectionImplementation: Removed .META.,,1.1028785192 for tableName=.META. from cache because of
12/11/08 14:23:02 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is 172.25.98.65:11411
Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
Thu Nov 08 14:22:23 UTC 2012, org.apache.hadoop.hbase.client.ScannerCallable@39ed1b0b, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
How can there be such a mismatch?
On Nov 7, 2012, at 8:08 AM, Mesika, Asaf wrote:

> After restart, running hbck results in:
> 12/11/07 06:07:28 DEBUG client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@51b4a74b
>
> 12/11/07 06:07:28 INFO util.HBaseFsck: getHTableDescriptors == tableNames => []
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck.reportTablesInFlux(HBaseFsck.java:560)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:346)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
>
> HBase is version 0.94.0
>
>
> On Nov 7, 2012, at 7:28 AM, Mesika, Asaf wrote:
>
>> Hi,
>>
>> I have a cluster with 10 RS and 1 Master.
>> From some reason, any call to the HBase fails. I ran hbck and got the same failure:
>>
>> Tue Nov 06 13:27:07 UTC 2012, org.apache.hadoop.hbase.client.ScannerCallable@7d8a8ce2, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
>>
>> at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:196)
>> at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>> at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>> at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>> at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:77)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:61)
>> at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntries(HBaseFsck.java:2234)
>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:334)
>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
>> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
>>
>>
>> Anyone has any idea how to fix it without the obvious restart?
>>
>