Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # dev - [question about replication] how to apply delta from Master to Slave after crash ?


+
Demai Ni 2013-07-22, 23:42
+
Jean-Daniel Cryans 2013-07-26, 17:08
+
Demai Ni 2013-07-26, 17:44
+
Jean-Daniel Cryans 2013-07-26, 17:53
+
Demai Ni 2013-07-26, 18:50
+
Jean-Daniel Cryans 2013-07-26, 19:59
Copy link to this message
-
Re: [question about replication] how to apply delta from Master to Slave after crash ?
Jean-Daniel Cryans 2013-07-26, 20:08
https://issues.apache.org/jira/browse/HBASE-9047

On Fri, Jul 26, 2013 at 12:59 PM, Jean-Daniel Cryans
<[EMAIL PROTECTED]> wrote:
> I guess I didn't explain my ideas clearly.
>
> So, first, replication in HBase is master-push, so you don't want to
> reverse the process. It means that this tool needs to run on the
> master cluster.
>
> Then I don't think you need to specify a timestamp since the
> replication state is in ZK. Basically that tool we're talking about
> would be able to read the replication state of each master region
> server, finish replicating what's missing, and then clear that state
> in zookeeper.
>
> The code that handles replication does most of that already. Check
> ReplicationSourceManager and ReplicationSource. Basically when
> ReplicationSourceManager.init() is called, it will check all the
> queues in ZK and try to grab those that aren't attached to a region
> server. If the whole cluster is down, it will grab all of them.
>
> The beautiful thing here is that you could start that tool on all your
> machines and the load will be spread out, but that might not be a big
> concern if replication wasn't lagging since it would take a few
> seconds to finish replicating the missing data for each region server.
>
> I'll open a jira.
>
> J-D
>
> On Fri, Jul 26, 2013 at 11:50 AM, Demai Ni <[EMAIL PROTECTED]> wrote:
>> JD,
>>
>> yeah. that sounds what I will need to do. a tool like this
>> [slave_cluster]$tool_to_syncup master_ZKquorum table_name start_timestamp
>>
>> so two tasks for me:
>> 1) identify the start_timestamp
>> 2) write the tool_to_syncup which will reach to master_ZK, copy the HLOGs
>> from makster, replay the HLOGs on Slave.
>>
>> are you aware of some example code for the 2) task that I can leverage?
>> thanks
>>
>> Demai
+
Demai Ni 2013-07-26, 20:29
+
Jean-Daniel Cryans 2013-07-26, 20:31