Hi,

I am trying to understand in detail how HBase replication works.

First of all, I assume that it is required for replication to work
correct that all edits are replayed on the replica HBase cluster in the
same order as they were executed on the source HBase cluster. Correct?

If so, I am trying to understand how that is guaranteed.

I can see that this is trivially true by reading the edits in the HLog,
and using that as a source for replication.

However, what if a region is moved to another region server. Can we not
end up in the following sitation?

1) region A is originally hosted by region server X.
2) replication in region server X is replicating edits of region A. Say
that it is lagging behind a bit, so it has a number of edits still to do.
3) region A is moved to region server Y.
4) edits for region A arrive on region server Y, and replication on
region server Y starts replicating them
5) replication in region server X is still busy with some left over
edits from region A, so these are replicated out of order

So the question really is whether there is a mechanism to prevent the
replication source from reading edits in a HLog for a region that was
meanwhile already moved to another region server.

It could be that it has something to do with log splitting and recovery,
but I was under the assumption that HBase only splits logs in case of
recovery and/or master restart, and not in case of region moves.

I hope somebody can shed some light on this topic.

Thanks,
Jan

On Fri, Nov 9, 2012 at 5:13 AM, Jan Van Besien <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I am trying to understand in detail how HBase replication works.
>
> First of all, I assume that it is required for replication to work correct
> that all edits are replayed on the replica HBase cluster in the same order
> as they were executed on the source HBase cluster. Correct?
>
Not really.... as it just tails the WAL at each regionserver in the
source cluster and sends the edits to the destination cluster. The
keyvalues are not altered while sending to the slave, so even if they
reach out of order, they settle down there with  the same effect. They
are in order from a source cluster regionserver's perspective, not
from a Table instance perspective.

You can refer to blog posts about replication working.
http://hbase.apache.org/replication.html

Himanshu

> If so, I am trying to understand how that is guara
nteed.
>
> I can see that this is trivially true by reading the edits in the HLog, and
> using that as a source for replication.
>
> However, what if a region is moved to another region server. Can we not end
> up in the following sitation?
>
> 1) region A is originally hosted by region server X.
> 2) replication in region server X is replicating edits of region A. Say that
> it is lagging behind a bit, so it has a number of edits still to do.
> 3) region A is moved to region server Y.
> 4) edits for region A arrive on region server Y, and replication on region
> server Y starts replicating them
> 5) replication in region server X is still busy with some left over edits
> from region A, so these are replicated out of order
>
> So the question really is whether there is a mechanism to prevent the
> replication source from reading edits in a HLog for a region that was
> meanwhile already moved to another region server.
>
> It could be that it has something to do with log splitting and recovery, but
> I was under the assumption that HBase only splits logs in case of recovery
> and/or master restart, and not in case of region moves.
>
> I hope somebody can shed some light on this topic.
>
> Thanks,
> Jan

On 11/09/2012 07:01 PM, Himanshu Vashishtha wrote:
>> First of all, I assume that it is required for replication to work correct
>> that all edits are replayed on the replica HBase cluster in the same order
>> as they were executed on the source HBase cluster. Correct?
>>
> Not really.... as it just tails the WAL at each regionserver in the
> source cluster and sends the edits to the destination cluster. The
> keyvalues are not altered while sending to the slave, so even if they
> reach out of order, they settle down there with  the same effect. They
> are in order from a source cluster regionserver's perspective, not
> from a Table instance perspective.

Thanks. I overlooked the way HBase deals with versions of key value pairs.

It does however still mean that it is possible to see rows on the
replica in a state that never occurred on the original HBase cluster, in
case put (or even delete) operations are not replicated in original
order and a client is reading the "latest" state.

So it is "eventually consistent", but when it is "not yet consistent"
(replication out of order and ongoing), there is no way on the replica
to know this..

thanks,
Jan

ps: sorry for the email duplicates of my original question.

Hi all,

On 11/12/2012 10:15 AM, Jan Van Besien wrote:
> It does however still mean that it is possible to see rows on the
> replica in a state that never occurred on the original HBase cluster, in
> case put (or even delete) operations are not replicated in original
> order and a client is reading the "latest" state.

Can anybody confirm or deny my above statement? I would like to know
whether my understanding of the replication feature is correct or not.
Thanks.
Additionally, I do think there might still be another problem with the
correctness guarantees of replication when edits arrive on the replica
"out of order" (which can happen AFAIK when a region server moves and
thus new edits for the same region are for a while potentially
replicated in parallel with older entries of that region).

Say that a put was followed by a delete, but the edits get reordered on
the replica, thus first issuing the delete, then the put.

Normally, it should be ok. The delete marker will be written on the
replica and then the put, but the versions of the key/values in the put
are older, thus the delete marker will still "win" when reading back the
data.

However, what if in between the delete and the put, a major compaction
happens. This major compaction cleans up the delete marker, so in that
case, the put will "win". Is this a situation in which the replica will
be incorrect?

Does this make sense? Or am I missing something completely?

Thanks in advance,
Jan

Yes. If there are failures in the source cluster the replicated data might be delivered out of order.
Note that you will never see partial rows applied; replication will enforce HBase's ACID guarantee here,
but rows might be delivered out of order, which means the ordering of deletes and puts might change, and hence lead to *temporary* visibility of deleted data. Eventually the state will be correct, though.
-- Lars

________________________________
 From: Jan Van Besien <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Wednesday, November 14, 2012 11:54 PM
Subject: Re: HBase replication: "in order semantics"
 
Hi all,

On 11/12/2012 10:15 AM, Jan Van Besien wrote:
> It does however still mean that it is possible to see rows on the
> replica in a state that never occurred on the original HBase cluster, in
> case put (or even delete) operations are not replicated in original
> order and a client is reading the "latest" state.

Can anybody confirm or deny my above statement? I would like to know whether my understanding of the replication feature is correct or not. Thanks.
Additionally, I do think there might still be another problem with the correctness guarantees of replication when edits arrive on the replica "out of order" (which can happen AFAIK when a region server moves and thus new edits for the same region are for a while potentially replicated in parallel with older entries of that region).

Say that a put was followed by a delete, but the edits get reordered on the replica, thus first issuing the delete, then the put.

Normally, it should be ok. The delete marker will be written on the replica and then the put, but the versions of the key/values in the put are older, thus the delete marker will still "win" when reading back the data.

However, what if in between the delete and the put, a major compaction happens. This major compaction cleans up the delete marker, so in that case, the put will "win". Is this a situation in which the replica will be incorrect?

Does this make sense? Or am I missing something completely?

Thanks in advance,
Jan

On 11/15/2012 06:45 PM, lars hofhansl wrote:
> Yes. If there are failures in the source cluster the replicated data might be delivered out of order.

ok.

> Note that you will never see partial rows applied; replication will enforce HBase's ACID guarantee here,

ok.

> but rows might be delivered out of order, which means the ordering of deletes and puts might change, and hence lead to *temporary* visibility of deleted data. Eventually the state will be correct, though.

I understand that there are situations in which the reordering results
in temporary visibility of deleted data. But additionally I also think
that there are (unlikely) situations which can lead to permanent
visibility of deleted data on the replica.

The situation I think of is:
- put followed by delete on source cluster
- reordering resulting in delete followed by put on replica
- delete gets executed on replica
- a major compaction happens on the replica (thus "really" deleting)
- put gets executed on the replica (thus permanently staying there)

Is there an error in my reasoning?

thanks,
Jan

You are right on the last scenario. Deleted data might permanently resurface in that case.

HBASE-4721 is an attempt to deal with that (mostly a hack, IMHO).
-- Lars

________________________________
 From: Jan Van Besien <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Friday, November 16, 2012 3:18 AM
Subject: Re: HBase replication: "in order semantics"
 
On 11/15/2012 06:45 PM, lars hofhansl wrote:
> Yes. If there are failures in the source cluster the replicated data might be delivered out of order.

ok.

> Note that you will never see partial rows applied; replication will enforce HBase's ACID guarantee here,

ok.

> but rows might be delivered out of order, which means the ordering of deletes and puts might change, and hence lead to *temporary* visibility of deleted data. Eventually the state will be correct, though.

I understand that there are situations in which the reordering results in temporary visibility of deleted data. But additionally I also think that there are (unlikely) situations which can lead to permanent visibility of deleted data on the replica.

The situation I think of is:
- put followed by delete on source cluster
- reordering resulting in delete followed by put on replica
- delete gets executed on replica
- a major compaction happens on the replica (thus "really" deleting)
- put gets executed on the replica (thus permanently staying there)

Is there an error in my reasoning?

thanks,
Jan