-Re: [question about replication] how to apply delta from Master to Slave after crash ?
Jean-Daniel Cryans 2013-07-26, 20:31
For more information on how replication works, please see
Also regarding the tool, I don't have time right now to work on it, so
don't expect something soon unless you want to work on it :)
Have a good weekend,
On Fri, Jul 26, 2013 at 1:29 PM, Demai Ni <[EMAIL PROTECTED]> wrote:
> JD, thanks for the jira and further explanation. for some reason, I was
> always thinking about 'pull' while considering the solution. Certainly
> 'push' the natural way to address this on Apache's hbase. I got your
> points now. Appreciate it... Demai
> On Fri, Jul 26, 2013 at 1:08 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:
>> On Fri, Jul 26, 2013 at 12:59 PM, Jean-Daniel Cryans
>> <[EMAIL PROTECTED]> wrote:
>> > I guess I didn't explain my ideas clearly.
>> > So, first, replication in HBase is master-push, so you don't want to
>> > reverse the process. It means that this tool needs to run on the
>> > master cluster.
>> > Then I don't think you need to specify a timestamp since the
>> > replication state is in ZK. Basically that tool we're talking about
>> > would be able to read the replication state of each master region
>> > server, finish replicating what's missing, and then clear that state
>> > in zookeeper.
>> > The code that handles replication does most of that already. Check
>> > ReplicationSourceManager and ReplicationSource. Basically when
>> > ReplicationSourceManager.init() is called, it will check all the
>> > queues in ZK and try to grab those that aren't attached to a region
>> > server. If the whole cluster is down, it will grab all of them.
>> > The beautiful thing here is that you could start that tool on all your
>> > machines and the load will be spread out, but that might not be a big
>> > concern if replication wasn't lagging since it would take a few
>> > seconds to finish replicating the missing data for each region server.
>> > I'll open a jira.
>> > J-D
>> > On Fri, Jul 26, 2013 at 11:50 AM, Demai Ni <[EMAIL PROTECTED]> wrote:
>> >> JD,
>> >> yeah. that sounds what I will need to do. a tool like this
>> >> [slave_cluster]$tool_to_syncup master_ZKquorum table_name
>> >> so two tasks for me:
>> >> 1) identify the start_timestamp
>> >> 2) write the tool_to_syncup which will reach to master_ZK, copy the
>> >> from makster, replay the HLOGs on Slave.
>> >> are you aware of some example code for the 2) task that I can leverage?
>> >> thanks
>> >> Demai