Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Region is not online: .META.,,1


+
Mesika, Asaf 2012-11-07, 05:28
+
Mesika, Asaf 2012-11-07, 06:08
Copy link to this message
-
Re: Region is not online: .META.,,1
More updates in case it can help:

In the UI I get the following regions mapping for .META.:
Table Regions

Name Region Server Start Key End Key Requests
.META.,,1.1028785192 172.25.98.66:11401 -
When I run 'hbase hbck' I fail, with the following error which indicates is thinks this region is placed in a different server (65 and not 66)

12/11/08 14:23:02 DEBUG client.HConnectionManager$HConnectionImplementation: Removed .META.,,1.1028785192 for tableName=.META. from cache because of
12/11/08 14:23:02 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is 172.25.98.65:11411
Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions:
Thu Nov 08 14:22:23 UTC 2012, org.apache.hadoop.hbase.client.ScannerCallable@39ed1b0b, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
How can there be such a mismatch?
On Nov 7, 2012, at 8:08 AM, Mesika, Asaf wrote:

> After restart, running hbck results in:
> 12/11/07 06:07:28 DEBUG client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@51b4a74b
>
> 12/11/07 06:07:28 INFO util.HBaseFsck: getHTableDescriptors == tableNames => []
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck.reportTablesInFlux(HBaseFsck.java:560)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:346)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
>
> HBase is version 0.94.0
>
>
> On Nov 7, 2012, at 7:28 AM, Mesika, Asaf wrote:
>
>> Hi,
>>
>> I have a cluster with 10 RS and 1 Master.
>> From some reason, any call to the HBase fails. I ran hbck and got the same failure:
>>
>> Tue Nov 06 13:27:07 UTC 2012, org.apache.hadoop.hbase.client.ScannerCallable@7d8a8ce2, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
>>
>> at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:196)
>> at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
>> at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
>> at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
>> at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:77)
>> at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:61)
>> at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntries(HBaseFsck.java:2234)
>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:334)
>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
>> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
>>
>>
>> Anyone has any idea how to fix it without the obvious restart?
>>
>

+
Leonid Fedotov 2012-11-07, 15:43
+
Mesika, Asaf 2012-11-08, 12:23
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB