Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Never ending transtionning regions.


Copy link to this message
-
Re: Never ending transtionning regions.
Kevin O'dell 2013-02-23, 21:16
JM,

  How are you doing today?  Right before the file does not exist should be
another path.  Can you let me know if in that path there are a pointers
from a split to 2ebfef593a3d715b59b85670909182c9?  The directory may
already exist.  I have seen this a couple times now and am trying to ferret
out a root cause to open a JIRA with.  I suspect we have a split code bug
in .92+

On Sat, Feb 23, 2013 at 4:10 PM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I have 2 regions transitionning from servers to servers for 15 minutes now.
>
> I have nothing in the master logs about those 2 regions but on the region
> server logs I have some files notfound2013-02-23 16:02:07,347 ERROR
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
> of region=entry,theykey,1361651769136.6dd77bc9ff91e0e6d413f74e670ab435.,
> starting to roll back the global memstore size.
> java.io.IOException: java.io.IOException: java.io.FileNotFoundException:
> File does not exist:
>
> /hbase/entry/2ebfef593a3d715b59b85670909182c9/a/62b0aae45d59408dbcfc513954efabc7
>     at
>
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:597)
>     at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:510)
>     at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4177)
>     at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4125)
>     at
>
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:328)
>     at
>
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:100)
>     at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
>     at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>     at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: java.io.FileNotFoundException: File does
> not exist:
>
> /hbase/entry/2ebfef593a3d715b59b85670909182c9/a/62b0aae45d59408dbcfc513954efabc7
>     at
> org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:433)
>     at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:240)
>     at
>
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3141)
>     at
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:572)
>     at
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:570)
>     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>     at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>     ... 3 more
> Caused by: java.io.FileNotFoundException: File does not exist:
>
> /hbase/entry/2ebfef593a3d715b59b85670909182c9/a/62b0aae45d59408dbcfc513954efabc7
>     at
>
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1843)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1834)
>     at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:578)
>     at
>
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:154)
>     at
> org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:108)
>     at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
>     at
>
> org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:573)
>     at
>
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1261)
>     at
>
> org.apache.hadoop.hbase.io.HalfStoreFileReader.<init>(HalfStoreFileReader.java:70)
>     at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:508)
>     at
>
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:603)

Kevin O'Dell
Customer Operations Engineer, Cloudera