Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> 0.96.0 keep logging "Memstore is above high water mark"


Copy link to this message
-
Re: 0.96.0 keep logging "Memstore is above high water mark"
I'm getting multi lines in the logs. I did not count them but sound like
it's one per client.

I will re-run the test and wc the lines to see if it's really one for one...

JM
2013/9/3 Jean-Daniel Cryans <[EMAIL PROTECTED]>

> The reason I was asking is if you used --nomapred, all your clients would
> share the same connection so only one stack trace should be printed (when
> the connection is closed).
>
> Since you're running with MR, how many stack traces do you see? Still one?
> If so, is it a local job runner? I'm trying to understand how many
> connections you are using.
>
> In any case, it's a normal-ish stack trace.
>
> J-D
>
>
> On Tue, Sep 3, 2013 at 11:14 AM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
> > I was running a MR with 10 clients, so it's a MapRed one.
> >
> > Just retried with only 1 thread and got the same exception.
> >
> > So it's not making any difference.
> >
> >
> > 2013/9/3 Jean-Daniel Cryans <[EMAIL PROTECTED]>
> >
> > > Alright so your ZK is running inside the master and that's where you're
> > > seeing those lines, and they are normal if each thread has a different
> > > connection... Are you doing a --nomapred PE?
> > >
> > >
> > > On Tue, Sep 3, 2013 at 10:50 AM, Jean-Marc Spaggiari <
> > > [EMAIL PROTECTED]> wrote:
> > >
> > > > Oh, ok ;) I just un-packed the jar and ran it. So my config file is:
> > > > <configuration>
> > > > </configuration>
> > > >
> > > >
> > > >
> > > > 2013/9/3 Jean-Daniel Cryans <[EMAIL PROTECTED]>
> > > >
> > > > > On Tue, Sep 3, 2013 at 10:35 AM, Jean-Marc Spaggiari <
> > > > > [EMAIL PROTECTED]> wrote:
> > > > >
> > > > > > > Your 10 clients are disconnecting from ZK, you're letting HBase
> > > > manage
> > > > > > it?
> > > > > > Yep, I start PE from the command line, I don't expect to have to
> do
> > > > > > anything after that. So issue is on PE side?
> > > > > >
> > > > >
> > > > > Not really what I asked, I wanted to know if you set
> HBASE_MANAGES_ZK
> > > at
> > > > > all in hbase-env.
> > > > >
> > > > >
> > > > > >
> > > > > > > Please don't remove the warn. It is important for
> troubleshooting
> > > and
> > > > > > sizing.
> > > > > > Can you please tell more on how it helps to do sizing?
> Interested.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > JM
> > > > > >
> > > > > >
> > > > > > 2013/9/3 Kevin O'dell <[EMAIL PROTECTED]>
> > > > > >
> > > > > > > Please don't remove the warn. It is important for
> troubleshooting
> > > and
> > > > > > > sizing.
> > > > > > > On Sep 3, 2013 1:29 PM, "Jean-Daniel Cryans" <
> > [EMAIL PROTECTED]>
> > > > > > wrote:
> > > > > > >
> > > > > > > > On Mon, Sep 2, 2013 at 12:04 PM, Jean-Marc Spaggiari <
> > > > > > > > [EMAIL PROTECTED]> wrote:
> > > > > > > >
> > > > > > > > > While running PE with 10 clients, server keep logging:
> > > > > > > > > 2013-09-02 14:56:59,919 WARN
> >  [RpcServer.handler=5,port=44439]
> > > > > > > > > regionserver.MemStoreFlusher: Memstore is above high water
> > mark
> > > > and
> > > > > > > > > block 2362ms
> > > > > > > > > 2013-09-02 14:56:59,919 WARN
> >  [RpcServer.handler=18,port=44439]
> > > > > > > > > regionserver.MemStoreFlusher: Memstore is above high water
> > mark
> > > > and
> > > > > > > > > block 2363ms
> > > > > > > > >
> > > > > > > >
> > > > > > > > Yeah that was added in HBASE-6466, it helps tracing when the
> > > > handlers
> > > > > > are
> > > > > > > > blocked on the memstores, else you have to match the
> "Blocking
> > > > > updates"
> > > > > > > > with the "Unblocking updates" lines. I'd just be fine adding
> > the
> > > > time
> > > > > > > into
> > > > > > > > the "Unblocking updates" line and remove that WARN.
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Then when test is over:
> > > > > > > > > 2013-09-02 14:57:02,280 WARN
> > > > > > > > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> > > > server.NIOServerCnxn:
> > > > > > > > > caught end of stream exception
> > > > > > > >