-Re: Questions about HBase replication
Varun Sharma 2013-05-20, 22:48
Thanks JD for the response... I was just wondering if issues have ever been
seen with regards to moving over a large number of WAL(s) entirely from one
region server to another since that would double the replication related
load on the one server which takes over.
Another side question: After the WAL has been replicated - is it purged
immediately or soonish from the zookeeper ?
On Mon, May 20, 2013 at 9:57 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:
> On Mon, May 20, 2013 at 12:35 AM, Varun Sharma <[EMAIL PROTECTED]>
> > Hi Lars,
> > Thanks for the response.
> > Regarding #2 again, so if RS1 failed, then the following happens...
> > 1) RS2 takes over its logs...
> > 2) Master renames the log containing directory to have a -splitting in
> > path
> > 3) Does RS2 already know about the "-splitting" path ?
> It will look at all the possible locations. See
> > Also on a related note, was there a reason that we have all region
> > watching all other region server's queue of logs. Otherwise, couldn't the
> > master have done the reassignment of outstanding logs to other region
> > servers more fairly upon failure ?
> I think I did it like that because it was easier since the region
> server has to be told to grab the queue(s) anyway.
> > Thanks
> > Varun
> > On Sun, May 19, 2013 at 8:49 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> >> #1 yes
> >> #2 no
> >> :)
> >> Now, there are scenarios where inconsistencies can happen. The edits are
> >> not necessarily shipped in order when there are failures.
> >> So it is possible to have some Puts at T1 and some Deletes at T2 (T1 <
> >> T2), and end up with the deletes shipped first.
> >> Now imagine a compaction happens at the slave after the Deletes are
> >> shipped to the slave, but before the Puts are shipped... The Puts will
> >> reappear.
> >> -- Lars
> >> ________________________________
> >> From: Varun Sharma <[EMAIL PROTECTED]>
> >> To: [EMAIL PROTECTED]
> >> Sent: Sunday, May 19, 2013 12:13 PM
> >> Subject: Questions about HBase replication
> >> Hi,
> >> I have a couple of questions about HBase replication...
> >> 1) When we ship edits to slave cluster - do we retain the timestamps in
> >> edits - if we don't, I can imagine hitting some inconsistencies ?
> >> 2) When a region server fails, the master renames the directory
> >> WAL(s). Does this impact reading of those logs for replication ?
> >> Thanks
> >> Varun