Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Transferring events across data centers


+
Apoorva Gaurav 2013-02-02, 02:48
+
Jun Rao 2013-02-04, 05:26
+
Apoorva Gaurav 2013-02-04, 06:02
+
Jun Rao 2013-02-04, 16:45
+
Apoorva Gaurav 2013-02-04, 17:30
+
Jun Rao 2013-02-05, 00:53
+
Apoorva Gaurav 2013-02-04, 06:03
+
S Ahmed 2013-02-04, 14:37
Copy link to this message
-
RE: Transferring events across data centers
Apoorva Gaurav 2013-02-04, 15:04
Yes peak numbers are per second

-apoorva
From: S Ahmed
Sent: 2/4/2013 8:07 PM
To: [EMAIL PROTECTED]
Subject: Re: Transferring events across data centers
what do you mean by this:

>>Number of events should be around
>>~50 million a day, peak being ~5K events a day, data volume ~100GB a day,
>>peak being ~10MB a day.
How can it be 50 million a day, with peak of 5K a day?  do you mean per
second?

On Fri, Feb 1, 2013 at 9:47 PM, Apoorva Gaurav <[EMAIL PROTECTED]>wrote:

> Hello All,
>
> We are working on Kafka based event collection system. This needs to gather
> events from across data centers. Lets say all the events will be produced
> in DC1 while kafka brokers and consumers are lying in DC2. Round trip
> between DC1 and DC2 can be around ~80 ms. Number of events should be around
> ~50 million a day, peak being ~5K events a day, data volume ~100GB a day,
> peak being ~10MB a day. What is the best way to do it.
>
> --- Is keeping the producer is DC1 sending events to DC2 a good idea.
> --- Should my ZK quorum lie only in DC2 or should it spawn across both DC1
> and DC2.
> --- Will this problem be solved easily in version .8.0 through broker
> replication by keeping brokers in both DC1 and DC2.
>
> --
> Thanks & Regards,
> Apoorvave
>