Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # dev - 0.96.0 keep logging "Memstore is above high water mark"


+
Jean-Marc Spaggiari 2013-09-02, 19:04
+
Jean-Daniel Cryans 2013-09-03, 17:28
+
Kevin Odell 2013-09-03, 17:30
+
Jean-Marc Spaggiari 2013-09-03, 17:35
+
Jean-Daniel Cryans 2013-09-03, 17:47
+
Kevin Odell 2013-09-03, 17:53
+
Jean-Marc Spaggiari 2013-09-03, 17:50
+
Jean-Daniel Cryans 2013-09-03, 18:00
+
Jean-Marc Spaggiari 2013-09-03, 18:14
+
Jean-Daniel Cryans 2013-09-03, 18:38
+
Jean-Marc Spaggiari 2013-09-03, 20:57
+
Jean-Marc Spaggiari 2013-09-03, 21:08
+
Jean-Daniel Cryans 2013-09-03, 21:48
+
Jean-Marc Spaggiari 2013-09-03, 22:16
Copy link to this message
-
Re: 0.96.0 keep logging "Memstore is above high water mark"
Jean-Daniel Cryans 2013-09-03, 22:21
It's probably not disconnecting properly so the zookeeper server prints
that exception. FWIW if your ZK server was started outside of HBase, you
wouldn't see those traces in the master log.

J-D
On Tue, Sep 3, 2013 at 3:16 PM, Jean-Marc Spaggiari <[EMAIL PROTECTED]
> wrote:

> So are you saying that it's normal to see a ZK warning each time a client
> disconnect? Or it's because PE is not disconnecting correctly?
> Le 2013-09-03 17:48, "Jean-Daniel Cryans" <[EMAIL PROTECTED]> a écrit :
>
> > I ran the command line you are using, it actually creates 100 clients and
> > this is why you see ~100 of them disconnecting from zookeeper.
> >
> > J-D
> >
> >
> > On Tue, Sep 3, 2013 at 2:08 PM, Jean-Marc Spaggiari <
> > [EMAIL PROTECTED]
> > > wrote:
> >
> > > Way more than 10...
> > >
> > > rm logs/*
> > > bin/start-hbase.sh
> > > bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=4096
> > > randomWrite 10
> > > cat logs/hbase-jmspaggiari-master-t430s.log  | grep "Unable to read
> > > additional" | wc
> > >     102    1530   12852
> > >
> > >
> > >
> > > 2013/9/3 Jean-Marc Spaggiari <[EMAIL PROTECTED]>
> > >
> > > > I'm getting multi lines in the logs. I did not count them but sound
> > like
> > > > it's one per client.
> > > >
> > > > I will re-run the test and wc the lines to see if it's really one for
> > > > one...
> > > >
> > > > JM
> > > >
> > > >
> > > > 2013/9/3 Jean-Daniel Cryans <[EMAIL PROTECTED]>
> > > >
> > > >> The reason I was asking is if you used --nomapred, all your clients
> > > would
> > > >> share the same connection so only one stack trace should be printed
> > > (when
> > > >> the connection is closed).
> > > >>
> > > >> Since you're running with MR, how many stack traces do you see?
> Still
> > > one?
> > > >> If so, is it a local job runner? I'm trying to understand how many
> > > >> connections you are using.
> > > >>
> > > >> In any case, it's a normal-ish stack trace.
> > > >>
> > > >> J-D
> > > >>
> > > >>
> > > >> On Tue, Sep 3, 2013 at 11:14 AM, Jean-Marc Spaggiari <
> > > >> [EMAIL PROTECTED]> wrote:
> > > >>
> > > >> > I was running a MR with 10 clients, so it's a MapRed one.
> > > >> >
> > > >> > Just retried with only 1 thread and got the same exception.
> > > >> >
> > > >> > So it's not making any difference.
> > > >> >
> > > >> >
> > > >> > 2013/9/3 Jean-Daniel Cryans <[EMAIL PROTECTED]>
> > > >> >
> > > >> > > Alright so your ZK is running inside the master and that's where
> > > >> you're
> > > >> > > seeing those lines, and they are normal if each thread has a
> > > different
> > > >> > > connection... Are you doing a --nomapred PE?
> > > >> > >
> > > >> > >
> > > >> > > On Tue, Sep 3, 2013 at 10:50 AM, Jean-Marc Spaggiari <
> > > >> > > [EMAIL PROTECTED]> wrote:
> > > >> > >
> > > >> > > > Oh, ok ;) I just un-packed the jar and ran it. So my config
> file
> > > is:
> > > >> > > > <configuration>
> > > >> > > > </configuration>
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > 2013/9/3 Jean-Daniel Cryans <[EMAIL PROTECTED]>
> > > >> > > >
> > > >> > > > > On Tue, Sep 3, 2013 at 10:35 AM, Jean-Marc Spaggiari <
> > > >> > > > > [EMAIL PROTECTED]> wrote:
> > > >> > > > >
> > > >> > > > > > > Your 10 clients are disconnecting from ZK, you're
> letting
> > > >> HBase
> > > >> > > > manage
> > > >> > > > > > it?
> > > >> > > > > > Yep, I start PE from the command line, I don't expect to
> > have
> > > >> to do
> > > >> > > > > > anything after that. So issue is on PE side?
> > > >> > > > > >
> > > >> > > > >
> > > >> > > > > Not really what I asked, I wanted to know if you set
> > > >> HBASE_MANAGES_ZK
> > > >> > > at
> > > >> > > > > all in hbase-env.
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > >
> > > >> > > > > > > Please don't remove the warn. It is important for
> > > >> troubleshooting
> > > >> > > and
> > > >> > > > > > sizing.
> > > >> > > > > > Can you please tell more on how it helps to do sizing?
> > > >> Interested.
+
Jean-Marc Spaggiari 2013-09-04, 12:05
+
Jean-Daniel Cryans 2013-09-03, 17:45
+
Frank Chow 2013-09-04, 12:32