Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Merging regions issue


Copy link to this message
-
Re: Merging regions issue
When a region merge is failing because of HBASE-1212, it leaves the
system in inconsistency form. I have created HBASE-7287 to fix that
until HBASE-1212 is resolved.

2012/12/5, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
> Ok. Seems that I'm facing HBASE-1212....
>
> The only issue is that when the merge is failing, hbck become inconsistent.
>
> JM
>
> 2012/12/5, Jean-Marc Spaggiari <[EMAIL PROTECTED]>:
>> Hi everyone,
>>
>> Sorry, I did not figured I was only replying to Marcos ;)
>>
>> So here are more details about this issue.
>>
>> I'm using HBase 0.94.3 and Hadoop 1.0.3.
>>
>> The merge seems to be failing when there is to many merges done.
>>
>> I just gave it another try... Each time I'm doing a major_compact
>> before trying the merges. And hbck.
>>
>> Build the table with 4 regions and 1000 rows. Keys are 8 bytes long
>> and value is 512 bytes. Everything went well, and hbck is not
>> reporting additionnal errors.
>>
>> Another try with 16 regions and 10000 rows worked well too.
>>
>> Another try with 54 regions and 10000 rows but this one is not working
>> fine. I'm getting some errors and the hbck is giving that:
>> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
>> testtable in hdfs dir
>> hdfs://node3:9000/hbase/testtable/88203ca27c9beedb02004d93e7181f94!
>> It may be an invalid format or version file.  Treating as an orphaned
>> regiondir.
>> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
>> testtable in hdfs dir
>> hdfs://node3:9000/hbase/testtable/c359655af1e7beb8138123e8aed4c382!
>> It may be an invalid format or version file.  Treating as an orphaned
>> regiondir.
>> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
>> testtable in hdfs dir
>> hdfs://node3:9000/hbase/testtable/e537a847f8c5a549993001b2bb9c0102!
>> It may be an invalid format or version file.  Treating as an orphaned
>> regiondir.
>> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
>> testtable in hdfs dir
>> hdfs://node3:9000/hbase/testtable/ef558e802a90b493677b5c07325b12fd!
>> It may be an invalid format or version file.  Treating as an orphaned
>> regiondir.
>>
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://node3:9000/hbase/testtable/88203ca27c9beedb02004d93e7181f94,
>> deployed =>  } on HDFS, but not listed in META or deployed on any
>> region server
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://node3:9000/hbase/testtable/c359655af1e7beb8138123e8aed4c382,
>> deployed =>  } on HDFS, but not listed in META or deployed on any
>> region server
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://node3:9000/hbase/testtable/e537a847f8c5a549993001b2bb9c0102,
>> deployed =>  } on HDFS, but not listed in META or deployed on any
>> region server
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://node3:9000/hbase/testtable/ef558e802a90b493677b5c07325b12fd,
>> deployed =>  } on HDFS, but not listed in META or deployed on any
>> region server
>>
>> And below is the log from the merge application. After that it's
>> listing all the regions in the server (a lot).
>>
>> I'm not sure if it's the first occurance of the issue or not.
>>
>> Should I open a JIRA for that? It's difficult to reproduce because
>> it's not a fixed pattern, but I still can get is failing easily.
>>
>> I will activate the DEBUG logs on the HRegion class, and give it again
>> and again another try ;)
>>
>> Thanks,
>>
>> JM
>>
>> Merging
>> testtable,?\xEC\x1B\x90^\xDB\xC9\xA5,1354727393540.3dca6fdbf95546ac71f47403a047fd10.
>> with
>> testtable,?\xEC\xA0_\xCE+\xB7),1354727402292.a5a42e51f3414fc45d77c1759378b58e.
>> 12/12/05 12:18:56 INFO util.Merge: Verifying that file system is
>> available...
>> 12/12/05 12:18:56 INFO util.Merge: Verifying that HBase is not running...
>> 12/12/05 12:18:56 INFO zookeeper.ZooKeeper: Initiating client
>> connection, connectString=latitude:2181,cube:2181,node3:2181
>> sessionTimeout=180000 watcher=hconnection
>> 12/12/05 12:18:56 INFO zookeeper.ClientCnxn: Opening socket connection
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB