Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Questions about HBase replication


Copy link to this message
-
Re: Questions about HBase replication
Thanks JD for the response... I was just wondering if issues have ever been
seen with regards to moving over a large number of WAL(s) entirely from one
region server to another since that would double the replication related
load on the one server which takes over.

Another side question: After the WAL has been replicated - is it purged
immediately or soonish from the zookeeper ?

Thanks
Varun
On Mon, May 20, 2013 at 9:57 AM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> On Mon, May 20, 2013 at 12:35 AM, Varun Sharma <[EMAIL PROTECTED]>
> wrote:
> > Hi Lars,
> >
> > Thanks for the response.
> >
> > Regarding #2 again, so if RS1 failed, then the following happens...
> > 1) RS2 takes over its logs...
> > 2) Master renames the log containing directory to have a -splitting in
> the
> > path
> > 3) Does RS2 already know about the "-splitting" path ?
>
> It will look at all the possible locations. See
> ReplicationSource.openReader
>
> >
> > Also on a related note, was there a reason that we have all region
> servers
> > watching all other region server's queue of logs. Otherwise, couldn't the
> > master have done the reassignment of outstanding logs to other region
> > servers more fairly upon failure ?
>
> I think I did it like that because it was easier since the region
> server has to be told to grab the queue(s) anyway.
>
> >
> > Thanks
> > Varun
> >
> >
> > On Sun, May 19, 2013 at 8:49 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> >
> >> #1 yes
> >> #2 no
> >>
> >> :)
> >>
> >> Now, there are scenarios where inconsistencies can happen. The edits are
> >> not necessarily shipped in order when there are failures.
> >> So it is possible to have some Puts at T1 and some Deletes at T2 (T1 <
> >> T2), and end up with the deletes shipped first.
> >> Now imagine a compaction happens at the slave after the Deletes are
> >> shipped to the slave, but before the Puts are shipped... The Puts will
> >> reappear.
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ________________________________
> >>  From: Varun Sharma <[EMAIL PROTECTED]>
> >> To: [EMAIL PROTECTED]
> >> Sent: Sunday, May 19, 2013 12:13 PM
> >> Subject: Questions about HBase replication
> >>
> >>
> >> Hi,
> >>
> >> I have a couple of questions about HBase replication...
> >>
> >> 1) When we ship edits to slave cluster - do we retain the timestamps in
> the
> >> edits - if we don't, I can imagine hitting some inconsistencies ?
> >>
> >> 2) When a region server fails, the master renames the directory
> containing
> >> WAL(s). Does this impact reading of those logs for replication ?
> >>
> >> Thanks
> >> Varun
> >>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB