Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> 0.96.0 keep logging "Memstore is above high water mark"


Copy link to this message
-
Re: 0.96.0 keep logging "Memstore is above high water mark"
I was running a MR with 10 clients, so it's a MapRed one.

Just retried with only 1 thread and got the same exception.

So it's not making any difference.
2013/9/3 Jean-Daniel Cryans <[EMAIL PROTECTED]>

> Alright so your ZK is running inside the master and that's where you're
> seeing those lines, and they are normal if each thread has a different
> connection... Are you doing a --nomapred PE?
>
>
> On Tue, Sep 3, 2013 at 10:50 AM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
> > Oh, ok ;) I just un-packed the jar and ran it. So my config file is:
> > <configuration>
> > </configuration>
> >
> >
> >
> > 2013/9/3 Jean-Daniel Cryans <[EMAIL PROTECTED]>
> >
> > > On Tue, Sep 3, 2013 at 10:35 AM, Jean-Marc Spaggiari <
> > > [EMAIL PROTECTED]> wrote:
> > >
> > > > > Your 10 clients are disconnecting from ZK, you're letting HBase
> > manage
> > > > it?
> > > > Yep, I start PE from the command line, I don't expect to have to do
> > > > anything after that. So issue is on PE side?
> > > >
> > >
> > > Not really what I asked, I wanted to know if you set HBASE_MANAGES_ZK
> at
> > > all in hbase-env.
> > >
> > >
> > > >
> > > > > Please don't remove the warn. It is important for troubleshooting
> and
> > > > sizing.
> > > > Can you please tell more on how it helps to do sizing? Interested.
> > > >
> > > > Thanks,
> > > >
> > > > JM
> > > >
> > > >
> > > > 2013/9/3 Kevin O'dell <[EMAIL PROTECTED]>
> > > >
> > > > > Please don't remove the warn. It is important for troubleshooting
> and
> > > > > sizing.
> > > > > On Sep 3, 2013 1:29 PM, "Jean-Daniel Cryans" <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >
> > > > > > On Mon, Sep 2, 2013 at 12:04 PM, Jean-Marc Spaggiari <
> > > > > > [EMAIL PROTECTED]> wrote:
> > > > > >
> > > > > > > While running PE with 10 clients, server keep logging:
> > > > > > > 2013-09-02 14:56:59,919 WARN  [RpcServer.handler=5,port=44439]
> > > > > > > regionserver.MemStoreFlusher: Memstore is above high water mark
> > and
> > > > > > > block 2362ms
> > > > > > > 2013-09-02 14:56:59,919 WARN  [RpcServer.handler=18,port=44439]
> > > > > > > regionserver.MemStoreFlusher: Memstore is above high water mark
> > and
> > > > > > > block 2363ms
> > > > > > >
> > > > > >
> > > > > > Yeah that was added in HBASE-6466, it helps tracing when the
> > handlers
> > > > are
> > > > > > blocked on the memstores, else you have to match the "Blocking
> > > updates"
> > > > > > with the "Unblocking updates" lines. I'd just be fine adding the
> > time
> > > > > into
> > > > > > the "Unblocking updates" line and remove that WARN.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Then when test is over:
> > > > > > > 2013-09-02 14:57:02,280 WARN
> > > > > > > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> > server.NIOServerCnxn:
> > > > > > > caught end of stream exception
> > > > > > > EndOfStreamException: Unable to read additional data from
> client
> > > > > > > sessionid 0x140dfdfc6270044, likely client has closed socket
> > > > > > >     at
> > > > > > >
> > > >
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
> > > > > > >     at
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> > > > > > >     at java.lang.Thread.run(Thread.java:662)
> > > > > > >
> > > > > >
> > > > > > Your 10 clients are disconnecting from ZK, you're letting HBase
> > > manage
> > > > > it?
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > /hbase-0.96.0$ cat logs/hbase-jmspaggiari-master-t430s.log  |
> > grep
> > > > > > > "Memstore is above high water mark and block" | wc
> > > > > > >    3555   49770  514558
> > > > > > >
> > > > > > >
> > > > > > > /hbase-0.96.0$ cat logs/hbase-jmspaggiari-master-t430s.log  |
> > grep
> > > > > > > "Unable to read additional data from client sessionid" | wc
> > > > > > >     102    1530   12852
> > > > > > >
> > > > > > > I guess it's only because of PE, but is this something which