Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Never ending transtionning regions.


+
Jean-Marc Spaggiari 2013-02-23, 21:10
+
Kevin Odell 2013-02-23, 21:16
Copy link to this message
-
Re: Never ending transtionning regions.
Jean-Marc Spaggiari 2013-02-23, 23:14
Hi Kevin,

Thanks for taking the time to reply.

Here is a bigger extract of the logs. I don't see another path in the logs.

http://pastebin.com/uMxGyjKm

I can send you the entire log if you want (42Mo)

What I did is I merged many regions together, then altered the table to set
the max_filesize and started a major_compaction to get the table splitted.

To fix the issue I had to drop one working table, and ran -repair multiple
times. Now it's fixed, but I still have the logs.

I'm redoing all the steps I did. Many I will face the issue again. If I'm
able to reproduce, we might be able to figure where the issue is...

JM

2013/2/23 Kevin O'dell <[EMAIL PROTECTED]>

> JM,
>
>   How are you doing today?  Right before the file does not exist should be
> another path.  Can you let me know if in that path there are a pointers
> from a split to 2ebfef593a3d715b59b85670909182c9?  The directory may
> already exist.  I have seen this a couple times now and am trying to ferret
> out a root cause to open a JIRA with.  I suspect we have a split code bug
> in .92+
>
> On Sat, Feb 23, 2013 at 4:10 PM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > I have 2 regions transitionning from servers to servers for 15 minutes
> now.
> >
> > I have nothing in the master logs about those 2 regions but on the region
> > server logs I have some files notfound2013-02-23 16:02:07,347 ERROR
> > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
> open
> > of region=entry,theykey,1361651769136.6dd77bc9ff91e0e6d413f74e670ab435.,
> > starting to roll back the global memstore size.
> > java.io.IOException: java.io.IOException: java.io.FileNotFoundException:
> > File does not exist:
> >
> >
> /hbase/entry/2ebfef593a3d715b59b85670909182c9/a/62b0aae45d59408dbcfc513954efabc7
> >     at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:597)
> >     at
> > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:510)
> >     at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4177)
> >     at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4125)
> >     at
> >
> >
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:328)
> >     at
> >
> >
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:100)
> >     at
> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> >     at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >     at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     at java.lang.Thread.run(Thread.java:722)
> > Caused by: java.io.IOException: java.io.FileNotFoundException: File does
> > not exist:
> >
> >
> /hbase/entry/2ebfef593a3d715b59b85670909182c9/a/62b0aae45d59408dbcfc513954efabc7
> >     at
> > org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:433)
> >     at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:240)
> >     at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3141)
> >     at
> > org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:572)
> >     at
> > org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:570)
> >     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >     at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >     at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >     at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >     ... 3 more
> > Caused by: java.io.FileNotFoundException: File does not exist:
> >
> >
> /hbase/entry/2ebfef593a3d715b59b85670909182c9/a/62b0aae45d59408dbcfc513954efabc7
> >     at
> >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1843)
+
Kevin Odell 2013-02-23, 23:21
+
Jean-Marc Spaggiari 2013-02-24, 00:36
+
Kevin Odell 2013-02-24, 00:41