Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Transferring events across data centers


+
Apoorva Gaurav 2013-02-02, 02:48
+
Jun Rao 2013-02-04, 05:26
+
Apoorva Gaurav 2013-02-04, 06:02
+
Jun Rao 2013-02-04, 16:45
+
Apoorva Gaurav 2013-02-04, 17:30
+
Jun Rao 2013-02-05, 00:53
+
Apoorva Gaurav 2013-02-04, 06:03
Copy link to this message
-
Re: Transferring events across data centers
what do you mean by this:

>>Number of events should be around
>>~50 million a day, peak being ~5K events a day, data volume ~100GB a day,
>>peak being ~10MB a day.
How can it be 50 million a day, with peak of 5K a day?  do you mean per
second?

On Fri, Feb 1, 2013 at 9:47 PM, Apoorva Gaurav <[EMAIL PROTECTED]>wrote:

> Hello All,
>
> We are working on Kafka based event collection system. This needs to gather
> events from across data centers. Lets say all the events will be produced
> in DC1 while kafka brokers and consumers are lying in DC2. Round trip
> between DC1 and DC2 can be around ~80 ms. Number of events should be around
> ~50 million a day, peak being ~5K events a day, data volume ~100GB a day,
> peak being ~10MB a day. What is the best way to do it.
>
> --- Is keeping the producer is DC1 sending events to DC2 a good idea.
> --- Should my ZK quorum lie only in DC2 or should it spawn across both DC1
> and DC2.
> --- Will this problem be solved easily in version .8.0 through broker
> replication by keeping brokers in both DC1 and DC2.
>
> --
> Thanks & Regards,
> Apoorvave
>

 
+
Apoorva Gaurav 2013-02-04, 15:04
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB