Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - [Shadow Regions / Read Replicas ]


Copy link to this message
-
答复: 答复: [Shadow Regions / Read Replicas ]
谢良 2013-12-13, 05:47
Hi Enis,

Thanks for reply. I have realized that still need a read secondary node ability
to achive lower 99th or 99.9th percentile read latency, e.g. big GC on rs node.
And i have a idea that impletement this ability from hbase-client side, we could
issue a read request to slave cluster, that will make us:
1) warm up slave cluster, so more performance confidence to switch the traffic
to slave cluster if the current master cluster is suffering a breakdown or sth.
2) we could have a very real stress testing result:)
We could impletement several policy on read just similar with your design's.
3) this behiviour could be similar with the traditional RDBMS operation: write to
master, and read from slave or slave+master :)  making the system scaling to read.

The mainly shortcoming of above is it's suitable only if having a replication running.
But i still like it and indeed also have a plan to write sth to do a prototype testing
next week. I still keep the same concern like before: the cost is heavy if you
want to enable the read from the secondary RS in the same cluster, with block cache
warming up always to achive the lower read latency.

Sorry my talking probably is not about the mainly design point(HA for read), but focus
on latency related.

Thanks,
________________________________________
发件人: Enis Söztutar [[EMAIL PROTECTED]]
发送时间: 2013年12月10日 5:24
收件人: [EMAIL PROTECTED]
主题: Re: 答复: [Shadow Regions / Read Replicas ]

We are also proposing to implement HBASE-7509 as a part of this major
undertaking. HBASE-7509 will help with HBase in general (even if you are
not using HBASE-10070), and possibly some other hdfs clients.
HBASE-10070 will give you similar benefits to HBASE-7509 if your use case
needs that, but on the hbase layer which will sit on top of HBASE-7509.

Enis
On Sat, Dec 7, 2013 at 5:39 AM, 谢良 <[EMAIL PROTECTED]> wrote:

> For one advantage of this design(ability to do low latency reads with
> <20ms 99.9% latencies for stale reads), to me, i more prefer to hbase-7509
> solution, Since if you want to ganrantee similar high performance read
> ability in
> shadow regions, then you must let the shadow rs warmup the related hot
> blocks
> into block cache.(In deed, i have a similar worry with Vladimir).
> I tried to think how this design could beat hbase-7509 on cutting the
> latency tail,
> but no result still.
>
> Enis, could you share your thoughts on it? thanks
>
> Thanks,
>
> ________________________________________
> 发件人: Enis Söztutar [[EMAIL PROTECTED]]
> 发送时间: 2013年12月4日 6:18
> 收件人: [EMAIL PROTECTED]
> 主题: Re: [Shadow Regions / Read Replicas ]
>
> On Tue, Dec 3, 2013 at 12:31 PM, Vladimir Rodionov
> <[EMAIL PROTECTED]>wrote:
>
> > The downside:
> >
> > - Double/Triple memstore usage
> > - Increased block cache usage (effectively, block cache will have 50%
> > capacity may be less)
>
>
> These are covered at the tradeoff section at the design doc.
>
>
> >
> >
> These downsides are pretty serious ones. This will result:
> >
> > 1. in decreased overall performance due to decreased efficient block
> cache
> > size
> >
>
> You can elect to not fill up the block cache for secondary reads. It will
> be a configuration option, and a
> tradeoff you may or may not want to pay. Details are in the doc.
>
>
> >  2. In more frequent memstore flushes - this will affect compaction and
> > write tput.
> >
>
> More frequent flushes is not needed unless you are using region snapshots
> approach,
> and want to bound the lag better. It is a tradeoff between expected lag vs
> more
> write amplification.
>
>
> >
> > I do not believe that  HBase 'large' MTTR does not allow to meet 99% SLA.
> > of 10-20ms unless your RSs go down 2-3 times a day for several minutes
> each
> > time. You have to analyze first why are you having so frequent failures,
> > than fix the root source of the problem. Its possible to reduce
> 'detection'
> > phase in MTTR process to couple seconds either by using external beacon
> > process (as I suggested already) or by rewriting some code inside HBase