Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> 0.96.0 keep logging "Memstore is above high water mark"


Copy link to this message
-
Re: 0.96.0 keep logging "Memstore is above high water mark"
Way more than 10...

rm logs/*
bin/start-hbase.sh
bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=4096
randomWrite 10
cat logs/hbase-jmspaggiari-master-t430s.log  | grep "Unable to read
additional" | wc
    102    1530   12852

2013/9/3 Jean-Marc Spaggiari <[EMAIL PROTECTED]>

> I'm getting multi lines in the logs. I did not count them but sound like
> it's one per client.
>
> I will re-run the test and wc the lines to see if it's really one for
> one...
>
> JM
>
>
> 2013/9/3 Jean-Daniel Cryans <[EMAIL PROTECTED]>
>
>> The reason I was asking is if you used --nomapred, all your clients would
>> share the same connection so only one stack trace should be printed (when
>> the connection is closed).
>>
>> Since you're running with MR, how many stack traces do you see? Still one?
>> If so, is it a local job runner? I'm trying to understand how many
>> connections you are using.
>>
>> In any case, it's a normal-ish stack trace.
>>
>> J-D
>>
>>
>> On Tue, Sep 3, 2013 at 11:14 AM, Jean-Marc Spaggiari <
>> [EMAIL PROTECTED]> wrote:
>>
>> > I was running a MR with 10 clients, so it's a MapRed one.
>> >
>> > Just retried with only 1 thread and got the same exception.
>> >
>> > So it's not making any difference.
>> >
>> >
>> > 2013/9/3 Jean-Daniel Cryans <[EMAIL PROTECTED]>
>> >
>> > > Alright so your ZK is running inside the master and that's where
>> you're
>> > > seeing those lines, and they are normal if each thread has a different
>> > > connection... Are you doing a --nomapred PE?
>> > >
>> > >
>> > > On Tue, Sep 3, 2013 at 10:50 AM, Jean-Marc Spaggiari <
>> > > [EMAIL PROTECTED]> wrote:
>> > >
>> > > > Oh, ok ;) I just un-packed the jar and ran it. So my config file is:
>> > > > <configuration>
>> > > > </configuration>
>> > > >
>> > > >
>> > > >
>> > > > 2013/9/3 Jean-Daniel Cryans <[EMAIL PROTECTED]>
>> > > >
>> > > > > On Tue, Sep 3, 2013 at 10:35 AM, Jean-Marc Spaggiari <
>> > > > > [EMAIL PROTECTED]> wrote:
>> > > > >
>> > > > > > > Your 10 clients are disconnecting from ZK, you're letting
>> HBase
>> > > > manage
>> > > > > > it?
>> > > > > > Yep, I start PE from the command line, I don't expect to have
>> to do
>> > > > > > anything after that. So issue is on PE side?
>> > > > > >
>> > > > >
>> > > > > Not really what I asked, I wanted to know if you set
>> HBASE_MANAGES_ZK
>> > > at
>> > > > > all in hbase-env.
>> > > > >
>> > > > >
>> > > > > >
>> > > > > > > Please don't remove the warn. It is important for
>> troubleshooting
>> > > and
>> > > > > > sizing.
>> > > > > > Can you please tell more on how it helps to do sizing?
>> Interested.
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > > JM
>> > > > > >
>> > > > > >
>> > > > > > 2013/9/3 Kevin O'dell <[EMAIL PROTECTED]>
>> > > > > >
>> > > > > > > Please don't remove the warn. It is important for
>> troubleshooting
>> > > and
>> > > > > > > sizing.
>> > > > > > > On Sep 3, 2013 1:29 PM, "Jean-Daniel Cryans" <
>> > [EMAIL PROTECTED]>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > On Mon, Sep 2, 2013 at 12:04 PM, Jean-Marc Spaggiari <
>> > > > > > > > [EMAIL PROTECTED]> wrote:
>> > > > > > > >
>> > > > > > > > > While running PE with 10 clients, server keep logging:
>> > > > > > > > > 2013-09-02 14:56:59,919 WARN
>> >  [RpcServer.handler=5,port=44439]
>> > > > > > > > > regionserver.MemStoreFlusher: Memstore is above high water
>> > mark
>> > > > and
>> > > > > > > > > block 2362ms
>> > > > > > > > > 2013-09-02 14:56:59,919 WARN
>> >  [RpcServer.handler=18,port=44439]
>> > > > > > > > > regionserver.MemStoreFlusher: Memstore is above high water
>> > mark
>> > > > and
>> > > > > > > > > block 2363ms
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > > Yeah that was added in HBASE-6466, it helps tracing when the
>> > > > handlers
>> > > > > > are
>> > > > > > > > blocked on the memstores, else you have to match the
>> "Blocking
>> > > > > updates"
>> > > > > > > > with the "Unblocking updates" lines. I'd just be fine adding