Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> [Shadow Regions / Read Replicas ]


Copy link to this message
-
答复: [Shadow Regions / Read Replicas ]
For one advantage of this design(ability to do low latency reads with
<20ms 99.9% latencies for stale reads), to me, i more prefer to hbase-7509
solution, Since if you want to ganrantee similar high performance read ability in
shadow regions, then you must let the shadow rs warmup the related hot blocks
into block cache.(In deed, i have a similar worry with Vladimir).
I tried to think how this design could beat hbase-7509 on cutting the latency tail,
but no result still.

Enis, could you share your thoughts on it? thanks

Thanks,

________________________________________
发件人: Enis Söztutar [[EMAIL PROTECTED]]
发送时间: 2013年12月4日 6:18
收件人: [EMAIL PROTECTED]
主题: Re: [Shadow Regions / Read Replicas ]

On Tue, Dec 3, 2013 at 12:31 PM, Vladimir Rodionov
<[EMAIL PROTECTED]>wrote:

> The downside:
>
> - Double/Triple memstore usage
> - Increased block cache usage (effectively, block cache will have 50%
> capacity may be less)
These are covered at the tradeoff section at the design doc.
>
>
These downsides are pretty serious ones. This will result:
>
> 1. in decreased overall performance due to decreased efficient block cache
> size
>

You can elect to not fill up the block cache for secondary reads. It will
be a configuration option, and a
tradeoff you may or may not want to pay. Details are in the doc.
>  2. In more frequent memstore flushes - this will affect compaction and
> write tput.
>

More frequent flushes is not needed unless you are using region snapshots
approach,
and want to bound the lag better. It is a tradeoff between expected lag vs
more
write amplification.
>
> I do not believe that  HBase 'large' MTTR does not allow to meet 99% SLA.
> of 10-20ms unless your RSs go down 2-3 times a day for several minutes each
> time. You have to analyze first why are you having so frequent failures,
> than fix the root source of the problem. Its possible to reduce 'detection'
> phase in MTTR process to couple seconds either by using external beacon
> process (as I suggested already) or by rewriting some code inside HBase and
> NameNode to move all data out from Java heap to off-heap and reducing
> GC-induced timeouts from 30 sec to 1-2 sec max. Its tough, but doable. The
> result: you will decrease MTTR by 50% at least w/o sacrificing the overall
> cluster performance.
>
> I think, its RS and NN large heaps   and frequent s-t-w GC  activities
> prevents meeting strict SLA - not occasional server failures.
>

MTTR and this work is ortagonal. In a distributed system, you cannot
differentiate between
a process not responding because it is down or it is busy or network is
down, or whatnot. Having
a couple of seconds detection time is unrealistic. You will end up in a
very unstable state where
you will be failing servers all over the place. An external beacon also
cannot differentiate between
the main process not responding because it is busy, or it is down. What
happens why there is a temporary
network partition.

>
>
>
> On Tue, Dec 3, 2013 at 11:51 AM, Jonathan Hsieh <[EMAIL PROTECTED]> wrote:
>
> > To keep the discussion focused on the design goals, I'm going start
> > referring to enis and deveraj's eventually consistent read replicas as
> the
> > *read replica* design, and consistent fast read recovery mechanism based
> on
> > shadowing/tailing the wals as *shadow regions* or *shadow memstores*.
>  Can
> > we agree on nomenclature?
> >
> >
> > On Tue, Dec 3, 2013 at 11:07 AM, Enis Söztutar <[EMAIL PROTECTED]> wrote:
> >
> > > Thanks Jon for bringing this to dev@.
> > >
> > >
> > > On Mon, Dec 2, 2013 at 10:01 PM, Jonathan Hsieh <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Fundamentally, I'd prefer focusing on making HBase "HBasier" instead
> of
> > > > tackling a feature that other systems architecturally can do better
> > > > (inconsistent reads).   I consider consistent reads/writes being one
> of
> > > > HBase's defining features. That said, I think read replicas makes
> sense
> > > and
> > > > is a nice feature to have.