Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Surprisingly high network traffic between kafka servers


Copy link to this message
-
Re: Surprisingly high network traffic between kafka servers
Guozhang Wang 2014-02-05, 05:26
Hi Carl,

For each partition the follower will also fetch data from the leader
replica, even if there is no new data in the leader replicas.

One thing you can try to increase replica.fetch.wait.max.ms (default value
500ms) so that the followers's fetching request frequency to the leader can
be reduced, and see if that has some effect on the traffic.

Guozhang
On Tue, Feb 4, 2014 at 8:46 PM, Carl Lerche <[EMAIL PROTECTED]> wrote:

> Hello,
>
> I'm running a 0.8.0 Kafka cluster of 3 servers. The service that it is
> for is not in full production yet, so the data written to cluster is
> minimal (seems to average between 100kb/s -> 300kb/s per server). I
> have configured Kafka to have a 3 replicas. I am noticing that each
> Kafka server is talking to all the others at a data rate of 40MB/s for
> each server (so, a total of 80MB/s for each server). This
> communication is constant.
>
> Is this normal? This seems like very strange behavior and I'm not
> exactly sure how to debug.
>
> Thanks,
> Carl
>

--
-- Guozhang