Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Merging regions issue


+
Jean-Marc Spaggiari 2012-12-04, 22:46
+
Jean-Marc Spaggiari 2012-12-04, 22:54
+
Marcos Ortiz 2012-12-05, 02:02
+
Marcos Ortiz 2012-12-05, 02:05
Copy link to this message
-
Re: Merging regions issue
Hi everyone,

Sorry, I did not figured I was only replying to Marcos ;)

So here are more details about this issue.

I'm using HBase 0.94.3 and Hadoop 1.0.3.

The merge seems to be failing when there is to many merges done.

I just gave it another try... Each time I'm doing a major_compact
before trying the merges. And hbck.

Build the table with 4 regions and 1000 rows. Keys are 8 bytes long
and value is 512 bytes. Everything went well, and hbck is not
reporting additionnal errors.

Another try with 16 regions and 10000 rows worked well too.

Another try with 54 regions and 10000 rows but this one is not working
fine. I'm getting some errors and the hbck is giving that:
ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
testtable in hdfs dir
hdfs://node3:9000/hbase/testtable/88203ca27c9beedb02004d93e7181f94!
It may be an invalid format or version file.  Treating as an orphaned
regiondir.
ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
testtable in hdfs dir
hdfs://node3:9000/hbase/testtable/c359655af1e7beb8138123e8aed4c382!
It may be an invalid format or version file.  Treating as an orphaned
regiondir.
ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
testtable in hdfs dir
hdfs://node3:9000/hbase/testtable/e537a847f8c5a549993001b2bb9c0102!
It may be an invalid format or version file.  Treating as an orphaned
regiondir.
ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
testtable in hdfs dir
hdfs://node3:9000/hbase/testtable/ef558e802a90b493677b5c07325b12fd!
It may be an invalid format or version file.  Treating as an orphaned
regiondir.

ERROR: Region { meta => null, hdfs =>
hdfs://node3:9000/hbase/testtable/88203ca27c9beedb02004d93e7181f94,
deployed =>  } on HDFS, but not listed in META or deployed on any
region server
ERROR: Region { meta => null, hdfs =>
hdfs://node3:9000/hbase/testtable/c359655af1e7beb8138123e8aed4c382,
deployed =>  } on HDFS, but not listed in META or deployed on any
region server
ERROR: Region { meta => null, hdfs =>
hdfs://node3:9000/hbase/testtable/e537a847f8c5a549993001b2bb9c0102,
deployed =>  } on HDFS, but not listed in META or deployed on any
region server
ERROR: Region { meta => null, hdfs =>
hdfs://node3:9000/hbase/testtable/ef558e802a90b493677b5c07325b12fd,
deployed =>  } on HDFS, but not listed in META or deployed on any
region server

And below is the log from the merge application. After that it's
listing all the regions in the server (a lot).

I'm not sure if it's the first occurance of the issue or not.

Should I open a JIRA for that? It's difficult to reproduce because
it's not a fixed pattern, but I still can get is failing easily.

I will activate the DEBUG logs on the HRegion class, and give it again
and again another try ;)

Thanks,

JM

Merging testtable,?\xEC\x1B\x90^\xDB\xC9\xA5,1354727393540.3dca6fdbf95546ac71f47403a047fd10.
with testtable,?\xEC\xA0_\xCE+\xB7),1354727402292.a5a42e51f3414fc45d77c1759378b58e.
12/12/05 12:18:56 INFO util.Merge: Verifying that file system is available...
12/12/05 12:18:56 INFO util.Merge: Verifying that HBase is not running...
12/12/05 12:18:56 INFO zookeeper.ZooKeeper: Initiating client
connection, connectString=latitude:2181,cube:2181,node3:2181
sessionTimeout=180000 watcher=hconnection
12/12/05 12:18:56 INFO zookeeper.ClientCnxn: Opening socket connection
to server /192.168.23.1:2181
12/12/05 12:18:56 INFO zookeeper.RecoverableZooKeeper: The identifier
of this process is 13131@node3
12/12/05 12:18:56 INFO client.ZooKeeperSaslClient: Client will not
SASL-authenticate because the default JAAS configuration section
'Client' could not be found. If you are not using SASL, you may ignore
this. On the other hand, if you expected SASL to work, please fix your
JAAS configuration.
12/12/05 12:18:56 INFO zookeeper.ClientCnxn: Socket connection
established to cube/192.168.23.1:2181, initiating session
12/12/05 12:18:56 INFO zookeeper.ClientCnxn: Session establishment
complete on server cube/192.168.23.1:2181, sessionid 0x13b6b25f2d900b6, negotiated timeout = 40000
12/12/05 12:18:56 INFO
client.HConnectionManager$HConnectionImplementation: ZooKeeper
available but no active master location found
12/12/05 12:18:56 INFO
client.HConnectionManager$HConnectionImplementation: getMaster attempt
0 of 1 failed; no more retrying.
org.apache.hadoop.hbase.MasterNotRunningException
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:674)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:110)
at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1733)
at org.apache.hadoop.hbase.util.Merge.run(Merge.java:94)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at MassMerger.mergeAllRegionsFromTable(MassMerger.java:81)
at MassMerger.main(MassMerger.java:120)
12/12/05 12:18:56 INFO
client.HConnectionManager$HConnectionImplementation: Closed zookeeper
sessionid=0x13b6b25f2d900b6
12/12/05 12:18:56 INFO zookeeper.ZooKeeper: Session: 0x13b6b25f2d900b6 closed
12/12/05 12:18:56 INFO zookeeper.ClientCnxn: EventThread shut down
12/12/05 12:18:56 INFO util.Merge: Merging regions
testtable,?\xEC\x1B\x90^\xDB\xC9\xA5,1354727393540.3dca6fdbf95546ac71f47403a047fd10.
and testtable,?\xEC\xA0_\xCE+\xB7),1354727402292.a5a42e51f3414fc45d77c1759378b58e.
in table testtable
12/12/05 12:18:56 INFO wal.HLog: FileSystem doesn't support getDefaultBlockSize
12/12/05 12:18:56 INFO wal.HLog: HLog configuration: blocksize=64 MB,
rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms
12/12/05 12:18:56 INFO wal.HLog:  for
/user/hbase/.logs_1354727936650/hlog.1354727936704
12/12/05 12:18:56 INFO wal.HLog: Using getNumCurrentReplicas--HDFS-826
12/12/05 12:18:56 INFO regionserver.HRegion: Setting up
tabledescriptor config now ...
12/12/05 12:18:56 INFO regionserver.Store: time to purge deletes set
to 0ms in store null
12/12/05 12:18:56 INFO regionserver.HRegion: Onlined
-ROO
+
Jean-Marc Spaggiari 2012-12-05, 18:46
+
Jean-Marc Spaggiari 2012-12-06, 00:30
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB