Kafka, mail # user - Transferring events across data centers - 2013-02-02, 02:48
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
Transferring events across data centers
Hello All,

We are working on Kafka based event collection system. This needs to gather
events from across data centers. Lets say all the events will be produced
in DC1 while kafka brokers and consumers are lying in DC2. Round trip
between DC1 and DC2 can be around ~80 ms. Number of events should be around
~50 million a day, peak being ~5K events a day, data volume ~100GB a day,
peak being ~10MB a day. What is the best way to do it.

and DC2.
replication by keeping brokers in both DC1 and DC2.

Thanks & Regards,

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB