Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Default value for Periodic Flusher


Copy link to this message
-
Re: Default value for Periodic Flusher
Yep, I obviously turned it off for my use case since MTTR is not a big
concern.
I just wanted to talk about the default value, not about my personal case
especially.
If you consider MTTR is the main objective, then I agree 1 hour is a decent
default value :)

Vladimir > you're right, I forgot to mention these points that made my
scenario even worse.

Thanks for sharing your opinions :)
On Wed, Dec 11, 2013 at 4:56 PM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> There is no perfect setting. For Adrien's usecase, this should be turned
> off. For some other usecases, this should be turned on.
>
> I'm -0 to turn it off. I will prefer people to usually have a good MTTR
> rather than less compactions. I'm not sure if 200gb region size is a common
> usecase.
>
>
> 2013/12/10 lars hofhansl <[EMAIL PROTECTED]>
>
> > Should we default it to 0 (off) then? (As Ted pointed out, you can turn
> > this off).
> > Are you not worried about MTTR when a RegionServer dies with 3GB of logs
> > to replay?
> >
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> >  From: Adrien Mogenet <[EMAIL PROTECTED]>
> > To: user <[EMAIL PROTECTED]>
> > Sent: Tuesday, December 10, 2013 2:16 PM
> > Subject: Re: Default value for Periodic Flusher
> >
> >
> > Hi guys,
> >
> > I've upgraded to 0.94.11. Here is my "worst-case scenario" :
> >
> > - let say each regionserver has 3 GB memstore
> > - let say compaction max filesize is ~200 GB, min. 2 files, max 10 files.
> > - let say memstore is growing "slowly" (1 GB / hour per RS)
> >
> > Then, automatically flushing every hour will lead into 1 GB storefiles,
> > being compacted into storefiles of 2 GB, 3 GB, 4.... up to 200 GB.
> > Sometimes, my write-load becomes very low, and periodic flusher will
> flush
> > perhaps 1 MB of data, it will trigger a minor compaction of hundreds
> > gigabytes + 1 MB; it seems to be lots of IO just to merge 1 MB of data.
> >
> > Previously (ie. lack of periodic flusher) memstore was creating 3 GB
> > storefiles, and thus creating (after minor compactions) 3 GB, 6 GB, 9
> GB...
> > up to 200 GB storefiles. And if memstore is growing slowly, it won't
> > generate small storefiles on HDFS. If think it looks like a more
> reasonable
> > IO-load, doesn't it?
> >
> > I deeply agree with Periodic Flusher relevance, but I don't think it's
> > suitable for everyone. Do you share my opinion wrt. my workload?
> >
> >
> > On Sun, Dec 8, 2013 at 10:36 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> > > Adrien:
> > > This config was introduced in 0.94.8
> > >
> > > Which release did you upgrade to ?
> > >
> > > As Jean-Marc said, telling us the issue (along with log snippet) would
> > > help.
> > >
> > > Cheers
> > >
> > >
> > > On Mon, Dec 9, 2013 at 1:26 AM, Jean-Marc Spaggiari <
> > > [EMAIL PROTECTED]
> > > > wrote:
> > >
> > > > Hi Adrien,
> > > >
> > > > What was the bad behavior you faced because of that? Maybe it's what
> > need
> > > > to be fixed more than the periodic flusher? Or put a bigger default
> > > value?
> > > >
> > > > JM
> > > >
> > > >
> > > > 2013/12/8 Adrien Mogenet <[EMAIL PROTECTED]>
> > > >
> > > > > Hi there,
> > > > >
> > > > > I'm wondering if the Periodic Flusher should be disabled by
> default?
> > > > >
> > > > > During a recent upgrade, I've noticed a strange behavior
> modification
> > > on
> > > > my
> > > > > servers, and it was due to this new feature, supplied with a "1
> hour"
> > > > > default value. I think upgrades should keep default behavior as
> close
> > > as
> > > > in
> > > > > previous versions. "By chance", logs were mentioning this "Periodic
> > > > > Flusher" but this could have taken much more time to debug
> otherwise
> > > :-)
> > > > >
> > > > > What are you thoughts guys?
> > > > > (Perhaps should be cc'ed to dev list?)
> > > > > --
> > > > > Adrien Mogenet
> > > > > http://www.borntosegfault.com
> >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Adrien Mogenet
> > http://www.borntosegfault.com

Adrien Mogenet
http://www.borntosegfault.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB