Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - state of replication in 0.8


Copy link to this message
-
Re: state of replication in 0.8
Jun Rao 2013-01-20, 18:25
Stan,

Quite a few people have been asking the same question. I created this wiki (
https://cwiki.apache.org/confluence/display/KAFKA/Changes+in+Kafka+0.8) to
document the significant changes in 0.8.

The replication design in 0.8 leveraged some ideas in the paper that you
mentioned, but is not the same. The key difference is that in 0.8, we
tolerate f-1 failures with f replicas. In contrast, typical Paxos-based
algorithms only tolerate f failures with 2*f + 1 replicas.

In terms of release schedule, we have seen some blocker issues when testing
0.8 on some production data at LinkedIn. Those blockers are being tracked
in
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+replication+development.
We plan to release a beta version once those blockers are resolved,
hopefully in the next few weeks.

Thanks,

Jun

On Sat, Jan 19, 2013 at 4:38 PM, Stan Rosenberg <[EMAIL PROTECTED]>wrote:

> Hi,
>
> Could someone please comment on the state of replication in 0.8, i.e.,
> what's implemented and what remains to be implemented?  On a first
> blush I see similarity between what's in the
> high-level documentation and Jun's paper, "Using Paxos to Build a
> Scalable, Consistent, and Highly Available Datastore".  I'd be
> especially grateful if someone could highlight the differences.
>
> Thanks!
>
> stan
>