-Re: [Shadow Regions / Read Replicas ] Wal per region?
Jonathan Hsieh 2013-12-03, 22:03
On Tue, Dec 3, 2013 at 11:42 AM, Enis Söztutar <[EMAIL PROTECTED]> wrote:
> On Mon, Dec 2, 2013 at 10:20 PM, Jonathan Hsieh <[EMAIL PROTECTED]> wrote:
> > > Deveraj:
> > > Jonathan Hsieh, WAL per region (WALpr) would give you the locality (and
> > hence HDFS short
> > > circuit) of reads if you were to couple it with the favored nodes. The
> > cost is of course more WAL
> > > files... In the current situation (no WALpr) it would create quite some
> > traffic cross machine, no?
> > I think we all agree that wal per region isn't efficient on today's
> > spinning hard drive world where we are limited to a relatively low budget
> > or seeks (though may be better in the future with SSD's).
> WALpr makes sense in fully SSD world and if hdfs had journaling for writes.
> I don't think anybody
> is working on this yet.
what do you mean by journaling for writes? do you mean where sync
operations update length at the nn on every call?
> Full SSD clusters are already in place (pinterest
> for example), so I
> think having WALpr as a pluggable implementation makes sense. HBase should
> work with both
> WAL-per-regionserver (or multi) or WAL-per-region.
> I agree here.
> > With this in mind, I actually I making the case that we would group the
> > the regions from RS-A onto the same set of preferred regions servers.
> > way we only need to have one or two other RS's tailing the RS.
> > So for example, if region X, Y and Z were on RS-A and its hlog, the
> > region memstores for X, Y, and Z would be assigned to the same one or two
> > other RSs. Ideally this would be where the HLog files replicas have
> > locality (helped by favored nodes/block affinity). Doing this, we hold
> > number of readers on the active hlogs to a constant number, do not add
> > new cross machine traffic (though tailing currently has costs on the NN).
> > One inefficiency we have is that if there is a single log per RS, we end
> > reading all the logs to tables that may not have the shadow feature
> > enabled. However, with HBase multi-wals coming, one strategy is to shard
> > wals to a number on the order of the number of disks on a machine (12-24
> > these days). I think the a wal per namespaces (this could be used to
> > a wal per table) of the hlog would make sense. This way of shardind the
> > hlog would reduce the amount of reading of irrelevant log entries on a
> > tailing scheme. It would have the added benefit of reducing the log
> > splitting work reducing MTTR and allowing for recovery priorities if the
> > primaries and shadows also go down. (this is an generalization of the
> > separate out the META into a separate log idea).
> > Jon.
> > --
> > // Jonathan Hsieh (shay)
> > // Software Engineer, Cloudera
> > // [EMAIL PROTECTED]
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// [EMAIL PROTECTED]