Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Questions about HBase replication


Copy link to this message
-
Re: Questions about HBase replication
Thanks JD for the response... I was just wondering if issues have ever been
seen with regards to moving over a large number of WAL(s) entirely from one
region server to another since that would double the replication related
load on the one server which takes over.

Another side question: After the WAL has been replicated - is it purged
immediately or soonish from the zookeeper ?

Thanks
Varun
On Mon, May 20, 2013 at 9:57 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> On Mon, May 20, 2013 at 12:35 AM, Varun Sharma <[EMAIL PROTECTED]>
> wrote:
> > Hi Lars,
> >
> > Thanks for the response.
> >
> > Regarding #2 again, so if RS1 failed, then the following happens...
> > 1) RS2 takes over its logs...
> > 2) Master renames the log containing directory to have a -splitting in
> the
> > path
> > 3) Does RS2 already know about the "-splitting" path ?
>
> It will look at all the possible locations. See
> ReplicationSource.openReader
>
> >
> > Also on a related note, was there a reason that we have all region
> servers
> > watching all other region server's queue of logs. Otherwise, couldn't the
> > master have done the reassignment of outstanding logs to other region
> > servers more fairly upon failure ?
>
> I think I did it like that because it was easier since the region
> server has to be told to grab the queue(s) anyway.
>
> >
> > Thanks
> > Varun
> >
> >
> > On Sun, May 19, 2013 at 8:49 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> >
> >> #1 yes
> >> #2 no
> >>
> >> :)
> >>
> >> Now, there are scenarios where inconsistencies can happen. The edits are
> >> not necessarily shipped in order when there are failures.
> >> So it is possible to have some Puts at T1 and some Deletes at T2 (T1 <
> >> T2), and end up with the deletes shipped first.
> >> Now imagine a compaction happens at the slave after the Deletes are
> >> shipped to the slave, but before the Puts are shipped... The Puts will
> >> reappear.
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ________________________________
> >>  From: Varun Sharma <[EMAIL PROTECTED]>
> >> To: [EMAIL PROTECTED]
> >> Sent: Sunday, May 19, 2013 12:13 PM
> >> Subject: Questions about HBase replication
> >>
> >>
> >> Hi,
> >>
> >> I have a couple of questions about HBase replication...
> >>
> >> 1) When we ship edits to slave cluster - do we retain the timestamps in
> the
> >> edits - if we don't, I can imagine hitting some inconsistencies ?
> >>
> >> 2) When a region server fails, the master renames the directory
> containing
> >> WAL(s). Does this impact reading of those logs for replication ?
> >>
> >> Thanks
> >> Varun
> >>
>