Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Never ending transtionning regions.


Copy link to this message
-
Never ending transtionning regions.
Hi,

I have 2 regions transitionning from servers to servers for 15 minutes now.

I have nothing in the master logs about those 2 regions but on the region
server logs I have some files notfound2013-02-23 16:02:07,347 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
of region=entry,theykey,1361651769136.6dd77bc9ff91e0e6d413f74e670ab435.,
starting to roll back the global memstore size.
java.io.IOException: java.io.IOException: java.io.FileNotFoundException:
File does not exist:
/hbase/entry/2ebfef593a3d715b59b85670909182c9/a/62b0aae45d59408dbcfc513954efabc7
    at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:597)
    at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:510)
    at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4177)
    at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4125)
    at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:328)
    at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:100)
    at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: java.io.FileNotFoundException: File does
not exist:
/hbase/entry/2ebfef593a3d715b59b85670909182c9/a/62b0aae45d59408dbcfc513954efabc7
    at
org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:433)
    at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:240)
    at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3141)
    at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:572)
    at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:570)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    ... 3 more
Caused by: java.io.FileNotFoundException: File does not exist:
/hbase/entry/2ebfef593a3d715b59b85670909182c9/a/62b0aae45d59408dbcfc513954efabc7
    at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1843)
    at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1834)
    at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:578)
    at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:154)
    at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:108)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
    at
org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:573)
    at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1261)
    at
org.apache.hadoop.hbase.io.HalfStoreFileReader.<init>(HalfStoreFileReader.java:70)
    at
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:508)
    at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:603)
    at org.apache.hadoop.hbase.regionserver.Store$1.call(Store.java:409)
    at org.apache.hadoop.hbase.regionserver.Store$1.call(Store.java:404)
    ... 8 more
2013-02-23 16:02:07,370 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x13d07ec012501fc Attempt to transition the unassigned
node for 6dd77bc9ff91e0e6d413f74e670ab435 from RS_ZK_REGION_OPENING to
RS_ZK_REGION_FAILED_OPEN failed, the node existed but was version 6586 not
the expected version 6585
If I try hbck -fix, this is bringing the master down:
2013-02-23 16:03:01,419 INFO org.apache.hadoop.hbase.master.HMaster:
BalanceSwitch=false
2013-02-23 16:03:03,067 FATAL org.apache.hadoop.hbase.master.HMaster:
Master server abort: loaded coprocessors are: []
2013-02-23 16:03:03,068 FATAL org.apache.hadoop.hbase.master.HMaster:
Unexpected state :
entry,thekey,1361651769136.6dd77bc9ff91e0e6d413f74e670ab435.
state=PENDING_OPEN, ts=1361653383067, server=node2,60020,1361653023303 ..
Cannot transit it to OFFLINE.
java.lang.IllegalStateException: Unexpected state :
entry,thekey,1361651769136.6dd77bc9ff91e0e6d413f74e670ab435.
state=PENDING_OPEN, ts=1361653383067, server=node2,60020,1361653023303 ..
Cannot transit it to OFFLINE.
    at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1813)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1658)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1423)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393)
    at
org.apache.hadoop.hbase.master.HMaster.assignRegion(HMaster.java:1740)
    at org.apache.hadoop.hbase.master.HMaster.assign(HMaster.java:1731)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
    at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
2013-02-23 16:03:03,069 INFO org.apache.hadoop.hbase.master.HMaster:
Aborting
2013-02-23 16:03:03,069 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60000
2013-02-23 16:03:03,069 INFO org.apache.hadoop.hbase.master.CatalogJanitor:
node3,60000,1361
+
Kevin Odell 2013-02-23, 21:16
+
Jean-Marc Spaggiari 2013-02-23, 23:14
+
Kevin Odell 2013-02-23, 23:21
+
Jean-Marc Spaggiari 2013-02-24, 00:36
+
Kevin Odell 2013-02-24, 00:41
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB