I don't think replication is ideal for creating single clusters spanning
DCs for at least a couple reasons: the replica assignment strategy is
currently not rack or DC-aware although that can be addressed by manually
creating topics. Also, network glitches and latencies which are more likely
in a cross-DC link could result in more frequent and prolonged periods of
under-replication, higher latencies in the controller-broker RPCs, etc. A
better approach would be to set up a mirror cluster - see
a few updates for 0.8)
Also, for your other question on the producer: a producer can only deliver
messages to the leader of a partition.
On Sun, Jun 23, 2013 at 6:10 PM, Mark Farnan <[EMAIL PROTECTED]> wrote:
> Is the replication factor system in Kafka 0.8 suitable for creating
> single clusters which span across data centers ? (up to 3)
> I am looking for a system where I don't loose messages, and can
> effectively 'fail over' to a different datacenter for processing if/when
> the primary goes down. If I read correctly, any message delivered to a
> Kafka broker, will be copied to its Replica/s. And a producer could
> deliver messages to any broker in the same replica set.
> Is that correct ?
> I am aware there are several zookeeper issues around multi DC support
> which I need to sort out, so this question is specific for the Kafka
> Note: My main consumer from Kafka will be STORM.