Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Region is not online: .META.,,1
Mesika, Asaf
2012-11-07, 05:28
Mesika, Asaf
2012-11-07, 06:08
Leonid Fedotov
2012-11-07, 15:43
Mesika, Asaf
2012-11-08, 12:23
+
Mesika, Asaf 2012-11-07, 05:28
+
Mesika, Asaf 2012-11-07, 06:08
Copy link to this message
-
Re: Region is not online: .META.,,1
Mesika, Asaf 2012-11-08, 14:31
More updates in case it can help:

In the UI I get the following regions mapping for .META.:
Table Regions

Name Region Server Start Key End Key Requests
.META.,,1.1028785192 172.25.98.66:11401 -
When I run 'hbase hbck' I fail, with the following error which indicates is thinks this region is placed in a different server (65 and not 66)

12/11/08 14:23:02 DEBUG client.HConnectionManager$HConnectionImplementation: Removed .META.,,1.1028785192 for tableName=.META. from cache because of
12/11/08 14:23:02 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is 172.25.98.65:11411
Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
Thu Nov 08 14:22:23 UTC 2012, org.apache.hadoop.hbase.client.ScannerCallable@39ed1b0b, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
How can there be such a mismatch?
On Nov 7, 2012, at 8:08 AM, Mesika, Asaf wrote:

> After restart, running hbck results in:
> 12/11/07 06:07:28 DEBUG client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@51b4a74b
>
> 12/11/07 06:07:28 INFO util.HBaseFsck: getHTableDescriptors == tableNames => []
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck.reportTablesInFlux(HBaseFsck.java:560)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:346)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
>
> HBase is version 0.94.0
>
>
> On Nov 7, 2012, at 7:28 AM, Mesika, Asaf wrote:
>
>> Hi,
>>
>> I have a cluster with 10 RS and 1 Master.
>> From some reason, any call to the HBase fails. I ran hbck and got the same failure:
>>
>> Tue Nov 06 13:27:07 UTC 2012, org.apache.hadoop.hbase.client.ScannerCallable@7d8a8ce2, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
>>
>> at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:196)
>> at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>> at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>> at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>> at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:77)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:61)
>> at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntries(HBaseFsck.java:2234)
>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:334)
>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
>> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
>>
>>
>> Anyone has any idea how to fix it without the obvious restart?
>>
>

+
Leonid Fedotov 2012-11-07, 15:43
+
Mesika, Asaf 2012-11-08, 12:23