Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Merging regions issue


+
Jean-Marc Spaggiari 2012-12-04, 22:46
+
Jean-Marc Spaggiari 2012-12-04, 22:54
+
Marcos Ortiz 2012-12-05, 02:02
+
Marcos Ortiz 2012-12-05, 02:05
+
Jean-Marc Spaggiari 2012-12-05, 17:30
+
Jean-Marc Spaggiari 2012-12-05, 18:46
Copy link to this message
-
Re: Merging regions issue
When a region merge is failing because of HBASE-1212, it leaves the
system in inconsistency form. I have created HBASE-7287 to fix that
until HBASE-1212 is resolved.

2012/12/5, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
> Ok. Seems that I'm facing HBASE-1212....
>
> The only issue is that when the merge is failing, hbck become inconsistent.
>
> JM
>
> 2012/12/5, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
>> Hi everyone,
>>
>> Sorry, I did not figured I was only replying to Marcos ;)
>>
>> So here are more details about this issue.
>>
>> I'm using HBase 0.94.3 and Hadoop 1.0.3.
>>
>> The merge seems to be failing when there is to many merges done.
>>
>> I just gave it another try... Each time I'm doing a major_compact
>> before trying the merges. And hbck.
>>
>> Build the table with 4 regions and 1000 rows. Keys are 8 bytes long
>> and value is 512 bytes. Everything went well, and hbck is not
>> reporting additionnal errors.
>>
>> Another try with 16 regions and 10000 rows worked well too.
>>
>> Another try with 54 regions and 10000 rows but this one is not working
>> fine. I'm getting some errors and the hbck is giving that:
>> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
>> testtable in hdfs dir
>> hdfs://node3:9000/hbase/testtable/88203ca27c9beedb02004d93e7181f94!
>> It may be an invalid format or version file.  Treating as an orphaned
>> regiondir.
>> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
>> testtable in hdfs dir
>> hdfs://node3:9000/hbase/testtable/c359655af1e7beb8138123e8aed4c382!
>> It may be an invalid format or version file.  Treating as an orphaned
>> regiondir.
>> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
>> testtable in hdfs dir
>> hdfs://node3:9000/hbase/testtable/e537a847f8c5a549993001b2bb9c0102!
>> It may be an invalid format or version file.  Treating as an orphaned
>> regiondir.
>> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
>> testtable in hdfs dir
>> hdfs://node3:9000/hbase/testtable/ef558e802a90b493677b5c07325b12fd!
>> It may be an invalid format or version file.  Treating as an orphaned
>> regiondir.
>>
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://node3:9000/hbase/testtable/88203ca27c9beedb02004d93e7181f94,
>> deployed =>  } on HDFS, but not listed in META or deployed on any
>> region server
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://node3:9000/hbase/testtable/c359655af1e7beb8138123e8aed4c382,
>> deployed =>  } on HDFS, but not listed in META or deployed on any
>> region server
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://node3:9000/hbase/testtable/e537a847f8c5a549993001b2bb9c0102,
>> deployed =>  } on HDFS, but not listed in META or deployed on any
>> region server
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://node3:9000/hbase/testtable/ef558e802a90b493677b5c07325b12fd,
>> deployed =>  } on HDFS, but not listed in META or deployed on any
>> region server
>>
>> And below is the log from the merge application. After that it's
>> listing all the regions in the server (a lot).
>>
>> I'm not sure if it's the first occurance of the issue or not.
>>
>> Should I open a JIRA for that? It's difficult to reproduce because
>> it's not a fixed pattern, but I still can get is failing easily.
>>
>> I will activate the DEBUG logs on the HRegion class, and give it again
>> and again another try ;)
>>
>> Thanks,
>>
>> JM
>>
>> Merging
>> testtable,?\xEC\x1B\x90^\xDB\xC9\xA5,1354727393540.3dca6fdbf95546ac71f47403a047fd10.
>> with
>> testtable,?\xEC\xA0_\xCE+\xB7),1354727402292.a5a42e51f3414fc45d77c1759378b58e.
>> 12/12/05 12:18:56 INFO util.Merge: Verifying that file system is
>> available...
>> 12/12/05 12:18:56 INFO util.Merge: Verifying that HBase is not running...
>> 12/12/05 12:18:56 INFO zookeeper.ZooKeeper: Initiating client
>> connection, connectString=latitude:2181,cube:2181,node3:2181
>> sessionTimeout=180000 watcher=hconnection
>> 12/12/05 12:18:56 INFO zookeeper.ClientCnxn: Opening socket connection