Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Never ending transtionning regions.


+
Jean-Marc Spaggiari 2013-02-23, 21:10
+
Kevin Odell 2013-02-23, 21:16
Copy link to this message
-
Re: Never ending transtionning regions.
Hi Kevin,

Thanks for taking the time to reply.

Here is a bigger extract of the logs. I don't see another path in the logs.

http://pastebin.com/uMxGyjKm

I can send you the entire log if you want (42Mo)

What I did is I merged many regions together, then altered the table to set
the max_filesize and started a major_compaction to get the table splitted.

To fix the issue I had to drop one working table, and ran -repair multiple
times. Now it's fixed, but I still have the logs.

I'm redoing all the steps I did. Many I will face the issue again. If I'm
able to reproduce, we might be able to figure where the issue is...

JM

2013/2/23 Kevin O'dell <[EMAIL PROTECTED]>

> JM,
>
>   How are you doing today?  Right before the file does not exist should be
> another path.  Can you let me know if in that path there are a pointers
> from a split to 2ebfef593a3d715b59b85670909182c9?  The directory may
> already exist.  I have seen this a couple times now and am trying to ferret
> out a root cause to open a JIRA with.  I suspect we have a split code bug
> in .92+
>
> On Sat, Feb 23, 2013 at 4:10 PM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > I have 2 regions transitionning from servers to servers for 15 minutes
> now.
> >
> > I have nothing in the master logs about those 2 regions but on the region
> > server logs I have some files notfound2013-02-23 16:02:07,347 ERROR
> > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
> open
> > of region=entry,theykey,1361651769136.6dd77bc9ff91e0e6d413f74e670ab435.,
> > starting to roll back the global memstore size.
> > java.io.IOException: java.io.IOException: java.io.FileNotFoundException:
> > File does not exist:
> >
> >
> /hbase/entry/2ebfef593a3d715b59b85670909182c9/a/62b0aae45d59408dbcfc513954efabc7
> >     at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:597)
> >     at
> > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:510)
> >     at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4177)
> >     at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4125)
> >     at
> >
> >
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:328)
> >     at
> >
> >
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:100)
> >     at
> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> >     at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >     at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >     at java.lang.Thread.run(Thread.java:722)
> > Caused by: java.io.IOException: java.io.FileNotFoundException: File does
> > not exist:
> >
> >
> /hbase/entry/2ebfef593a3d715b59b85670909182c9/a/62b0aae45d59408dbcfc513954efabc7
> >     at
> > org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:433)
> >     at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:240)
> >     at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3141)
> >     at
> > org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:572)
> >     at
> > org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:570)
> >     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >     at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >     at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >     at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >     ... 3 more
> > Caused by: java.io.FileNotFoundException: File does not exist:
> >
> >
> /hbase/entry/2ebfef593a3d715b59b85670909182c9/a/62b0aae45d59408dbcfc513954efabc7
> >     at
> >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1843)
+
Kevin Odell 2013-02-23, 23:21
+
Jean-Marc Spaggiari 2013-02-24, 00:36
+
Kevin Odell 2013-02-24, 00:41
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB