hi,all
use hbase-0.94.3,  We have encounted an error that results into
Regionserver offline.

Stack:

2012-12-25 19:25:04,654 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
10.1.1.34,60020,1356416242185: Uncaught exception in service thread
regionserver60020.cacheFlusher
java.lang.AssertionError: getSplitPoint() called on a region that can't
split!
        at
org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1984)
        at
org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:79)
        at
org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:5068)
        at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:415)
        at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
        at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:250)
        at java.lang.Thread.run(Thread.java:662)
2012-12-25 19:25:04,654 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
loaded coprocessors are: []
....
Can anyone meet these errors?
Cheers!
--
Bing Jiang
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science

Can you provide more log around the time this happened ?

Looks like the exception message should be enhanced with region name.

Cheers

On Dec 25, 2012, at 3:32 AM, Bing Jiang <[EMAIL PROTECTED]> wrote:

> hi,all
> use hbase-0.94.3,  We have encounted an error that results into
> Regionserver offline.
>
> Stack:
>
> 2012-12-25 19:25:04,654 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
> 10.1.1.34,60020,1356416242185: Uncaught exception in service thread
> regionserver60020.cacheFlusher
> java.lang.AssertionError: getSplitPoint() called on a region that can't
> split!
>        at
> org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1984)
>        at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:79)
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:5068)
>        at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:415)
>        at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
>        at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:250)
>        at java.lang.Thread.run(Thread.java:662)
> 2012-12-25 19:25:04,654 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
> loaded coprocessors are: []
> ....
> Can anyone meet these errors?
>
>
> Cheers!
> --
> Bing Jiang
> weibo: http://weibo.com/jiangbinglover
> BLOG: http://blog.sina.com.cn/jiangbinglover
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science

Oh, this seems interesting.
You have logs before this which says for which region the split started?
BTW are you using any custom split policy.

Regards
Ram

On Tue, Dec 25, 2012 at 7:43 PM, <[EMAIL PROTECTED]> wrote:

> Can you provide more log around the time this happened ?
>
> Looks like the exception message should be enhanced with region name.
>
> Cheers
>
> On Dec 25, 2012, at 3:32 AM, Bing Jiang <[EMAIL PROTECTED]> wrote:
>
> > hi,all
> > use hbase-0.94.3,  We have encounted an error that results into
> > Regionserver offline.
> >
> > Stack:
> >
> > 2012-12-25 19:25:04,654 FATAL
> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
> server
> > 10.1.1.34,60020,1356416242185: Uncaught exception in service thread
> > regionserver60020.cacheFlusher
> > java.lang.AssertionError: getSplitPoint() called on a region that can't
> > split!
> >        at
> > org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1984)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:79)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:5068)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:415)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:250)
> >        at java.lang.Thread.run(Thread.java:662)
> > 2012-12-25 19:25:04,654 FATAL
> > org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
> > loaded coprocessors are: []
> > ....
> > Can anyone meet these errors?
> >
> >
> > Cheers!
> > --
> > Bing Jiang
> > weibo: http://weibo.com/jiangbinglover
> > BLOG: http://blog.sina.com.cn/jiangbinglover
> > National Research Center for Intelligent Computing Systems
> > Institute of Computing technology
> > Graduate University of Chinese Academy of Science
>

By looking at the code, we have:
"Not splittable if we find a reference store file present in the store"...

Not really sure to understand what that mean :(

2012/12/25, ramkrishna vasudevan <[EMAIL PROTECTED]>:
> Oh, this seems interesting.
> You have logs before this which says for which region the split started?
> BTW are you using any custom split policy.
>
> Regards
> Ram
>
> On Tue, Dec 25, 2012 at 7:43 PM, <[EMAIL PROTECTED]> wrote:
>
>> Can you provide more log around the time this happened ?
>>
>> Looks like the exception message should be enhanced with region name.
>>
>> Cheers
>>
>> On Dec 25, 2012, at 3:32 AM, Bing Jiang <[EMAIL PROTECTED]> wrote:
>>
>> > hi,all
>> > use hbase-0.94.3,  We have encounted an error that results into
>> > Regionserver offline.
>> >
>> > Stack:
>> >
>> > 2012-12-25 19:25:04,654 FATAL
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
>> server
>> > 10.1.1.34,60020,1356416242185: Uncaught exception in service thread
>> > regionserver60020.cacheFlusher
>> > java.lang.AssertionError: getSplitPoint() called on a region that can't
>> > split!
>> >        at
>> > org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1984)
>> >        at
>> >
>> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:79)
>> >        at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:5068)
>> >        at
>> >
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:415)
>> >        at
>> >
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
>> >        at
>> >
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:250)
>> >        at java.lang.Thread.run(Thread.java:662)
>> > 2012-12-25 19:25:04,654 FATAL
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
>> > loaded coprocessors are: []
>> > ....
>> > Can anyone meet these errors?
>> >
>> >
>> > Cheers!
>> > --
>> > Bing Jiang
>> > weibo: http://weibo.com/jiangbinglover
>> > BLOG: http://blog.sina.com.cn/jiangbinglover
>> > National Research Center for Intelligent Computing Systems
>> > Institute of Computing technology
>> > Graduate University of Chinese Academy of Science
>>
>

The above error is an assertion error.  It comes from here
{code}
 // Should already be enforced by the split policy!
      assert !this.region.getRegionInfo().isMetaRegion();
{code}

So need to see this i feel.

Regards
Ram

On Tue, Dec 25, 2012 at 9:48 PM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> By looking at the code, we have:
> "Not splittable if we find a reference store file present in the store"...
>
> Not really sure to understand what that mean :(
>
> 2012/12/25, ramkrishna vasudevan <[EMAIL PROTECTED]>:
> > Oh, this seems interesting.
> > You have logs before this which says for which region the split started?
> > BTW are you using any custom split policy.
> >
> > Regards
> > Ram
> >
> > On Tue, Dec 25, 2012 at 7:43 PM, <[EMAIL PROTECTED]> wrote:
> >
> >> Can you provide more log around the time this happened ?
> >>
> >> Looks like the exception message should be enhanced with region name.
> >>
> >> Cheers
> >>
> >> On Dec 25, 2012, at 3:32 AM, Bing Jiang <[EMAIL PROTECTED]>
> wrote:
> >>
> >> > hi,all
> >> > use hbase-0.94.3,  We have encounted an error that results into
> >> > Regionserver offline.
> >> >
> >> > Stack:
> >> >
> >> > 2012-12-25 19:25:04,654 FATAL
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
> >> server
> >> > 10.1.1.34,60020,1356416242185: Uncaught exception in service thread
> >> > regionserver60020.cacheFlusher
> >> > java.lang.AssertionError: getSplitPoint() called on a region that
> can't
> >> > split!
> >> >        at
> >> >
> org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1984)
> >> >        at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:79)
> >> >        at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:5068)
> >> >        at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:415)
> >> >        at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
> >> >        at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:250)
> >> >        at java.lang.Thread.run(Thread.java:662)
> >> > 2012-12-25 19:25:04,654 FATAL
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer
> abort:
> >> > loaded coprocessors are: []
> >> > ....
> >> > Can anyone meet these errors?
> >> >
> >> >
> >> > Cheers!
> >> > --
> >> > Bing Jiang
> >> > weibo: http://weibo.com/jiangbinglover
> >> > BLOG: http://blog.sina.com.cn/jiangbinglover
> >> > National Research Center for Intelligent Computing Systems
> >> > Institute of Computing technology
> >> > Graduate University of Chinese Academy of Science
> >>
> >
>

Whoa. Unless a custom split policy was used here, this is bad.

________________________________
 From: ramkrishna vasudevan <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Tuesday, December 25, 2012 8:28 AM
Subject: Re: hbase-0.94.3 region cannot split , which results into regionserver offline？
 
The above error is an assertion error.  It comes from here
{code}
// Should already be enforced by the split policy!
      assert !this.region.getRegionInfo().isMetaRegion();
{code}

So need to see this i feel.

Regards
Ram

On Tue, Dec 25, 2012 at 9:48 PM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> By looking at the code, we have:
> "Not splittable if we find a reference store file present in the store"...
>
> Not really sure to understand what that mean :(
>
> 2012/12/25, ramkrishna vasudevan <[EMAIL PROTECTED]>:
> > Oh, this seems interesting.
> > You have logs before this which says for which region the split started?
> > BTW are you using any custom split policy.
> >
> > Regards
> > Ram
> >
> > On Tue, Dec 25, 2012 at 7:43 PM, <[EMAIL PROTECTED]> wrote:
> >
> >> Can you provide more log around the time this happened ?
> >>
> >> Looks like the exception message should be enhanced with region name.
> >>
> >> Cheers
> >>
> >> On Dec 25, 2012, at 3:32 AM, Bing Jiang <[EMAIL PROTECTED]>
> wrote:
> >>
> >> > hi,all
> >> > use hbase-0.94.3,  We have encounted an error that results into
> >> > Regionserver offline.
> >> >
> >> > Stack:
> >> >
> >> > 2012-12-25 19:25:04,654 FATAL
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
> >> server
> >> > 10.1.1.34,60020,1356416242185: Uncaught exception in service thread
> >> > regionserver60020.cacheFlusher
> >> > java.lang.AssertionError: getSplitPoint() called on a region that
> can't
> >> > split!
> >> >        at
> >> >
> org.apache.hadoop.hbase.regionserver.Store.getSplitPoint(Store.java:1984)
> >> >        at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:79)
> >> >        at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:5068)
> >> >        at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:415)
> >> >        at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
> >> >        at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:250)
> >> >        at java.lang.Thread.run(Thread.java:662)
> >> > 2012-12-25 19:25:04,654 FATAL
> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer
> abort:
> >> > loaded coprocessors are: []
> >> > ....
> >> > Can anyone meet these errors?
> >> >
> >> >
> >> > Cheers!
> >> > --
> >> > Bing Jiang
> >> > weibo: http://weibo.com/jiangbinglover
> >> > BLOG: http://blog.sina.com.cn/jiangbinglover
> >> > National Research Center for Intelligent Computing Systems
> >> > Institute of Computing technology
> >> > Graduate University of Chinese Academy of Science
> >>
> >
>