Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Default value for Periodic Flusher


Copy link to this message
-
Re: Default value for Periodic Flusher
Yep, I obviously turned it off for my use case since MTTR is not a big
concern.
I just wanted to talk about the default value, not about my personal case
especially.
If you consider MTTR is the main objective, then I agree 1 hour is a decent
default value :)

Vladimir > you're right, I forgot to mention these points that made my
scenario even worse.

Thanks for sharing your opinions :)
On Wed, Dec 11, 2013 at 4:56 PM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> There is no perfect setting. For Adrien's usecase, this should be turned
> off. For some other usecases, this should be turned on.
>
> I'm -0 to turn it off. I will prefer people to usually have a good MTTR
> rather than less compactions. I'm not sure if 200gb region size is a common
> usecase.
>
>
> 2013/12/10 lars hofhansl <[EMAIL PROTECTED]>
>
> > Should we default it to 0 (off) then? (As Ted pointed out, you can turn
> > this off).
> > Are you not worried about MTTR when a RegionServer dies with 3GB of logs
> > to replay?
> >
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> >  From: Adrien Mogenet <[EMAIL PROTECTED]>
> > To: user <[EMAIL PROTECTED]>
> > Sent: Tuesday, December 10, 2013 2:16 PM
> > Subject: Re: Default value for Periodic Flusher
> >
> >
> > Hi guys,
> >
> > I've upgraded to 0.94.11. Here is my "worst-case scenario" :
> >
> > - let say each regionserver has 3 GB memstore
> > - let say compaction max filesize is ~200 GB, min. 2 files, max 10 files.
> > - let say memstore is growing "slowly" (1 GB / hour per RS)
> >
> > Then, automatically flushing every hour will lead into 1 GB storefiles,
> > being compacted into storefiles of 2 GB, 3 GB, 4.... up to 200 GB.
> > Sometimes, my write-load becomes very low, and periodic flusher will
> flush
> > perhaps 1 MB of data, it will trigger a minor compaction of hundreds
> > gigabytes + 1 MB; it seems to be lots of IO just to merge 1 MB of data.
> >
> > Previously (ie. lack of periodic flusher) memstore was creating 3 GB
> > storefiles, and thus creating (after minor compactions) 3 GB, 6 GB, 9
> GB...
> > up to 200 GB storefiles. And if memstore is growing slowly, it won't
> > generate small storefiles on HDFS. If think it looks like a more
> reasonable
> > IO-load, doesn't it?
> >
> > I deeply agree with Periodic Flusher relevance, but I don't think it's
> > suitable for everyone. Do you share my opinion wrt. my workload?
> >
> >
> > On Sun, Dec 8, 2013 at 10:36 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> > > Adrien:
> > > This config was introduced in 0.94.8
> > >
> > > Which release did you upgrade to ?
> > >
> > > As Jean-Marc said, telling us the issue (along with log snippet) would
> > > help.
> > >
> > > Cheers
> > >
> > >
> > > On Mon, Dec 9, 2013 at 1:26 AM, Jean-Marc Spaggiari <
> > > [EMAIL PROTECTED]
> > > > wrote:
> > >
> > > > Hi Adrien,
> > > >
> > > > What was the bad behavior you faced because of that? Maybe it's what
> > need
> > > > to be fixed more than the periodic flusher? Or put a bigger default
> > > value?
> > > >
> > > > JM
> > > >
> > > >
> > > > 2013/12/8 Adrien Mogenet <[EMAIL PROTECTED]>
> > > >
> > > > > Hi there,
> > > > >
> > > > > I'm wondering if the Periodic Flusher should be disabled by
> default?
> > > > >
> > > > > During a recent upgrade, I've noticed a strange behavior
> modification
> > > on
> > > > my
> > > > > servers, and it was due to this new feature, supplied with a "1
> hour"
> > > > > default value. I think upgrades should keep default behavior as
> close
> > > as
> > > > in
> > > > > previous versions. "By chance", logs were mentioning this "Periodic
> > > > > Flusher" but this could have taken much more time to debug
> otherwise
> > > :-)
> > > > >
> > > > > What are you thoughts guys?
> > > > > (Perhaps should be cc'ed to dev list?)
> > > > > --
> > > > > Adrien Mogenet
> > > > > http://www.borntosegfault.com
> >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Adrien Mogenet
> > http://www.borntosegfault.com

Adrien Mogenet
http://www.borntosegfault.com