Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Surprisingly high network traffic between kafka servers


Copy link to this message
-
Re: Surprisingly high network traffic between kafka servers
Jay Kreps 2014-02-05, 20:31
Can you enable DEBUG logging in log4j and see what requests are coming in?

-Jay
On Tue, Feb 4, 2014 at 9:51 PM, Carl Lerche <[EMAIL PROTECTED]> wrote:

> Hi Jay,
>
> I do not believe that I have changed the replica.fetch.wait.max.ms
> setting. Here I have included the kafka config as well as a snapshot
> of jnettop from one of the servers.
>
> https://gist.github.com/carllerche/4f2cf0f0f6d1e891f482
>
> The bottom row (89.9K/s) is the producer (it lives on a Kafka server).
> The top two rows are Kafkas on other servers, you can see the combined
> throughput is ~80MB/s
>
> On Tue, Feb 4, 2014 at 9:36 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> > No this is not normal.
> >
> > Checking twice a second (using 500ms default) for new data shouldn't
> cause
> > high network traffic (that should be like < 1KB of overhead). I don't
> think
> > that explains things. Is it possible that setting has been overridden?
> >
> > -Jay
> >
> >
> > On Tue, Feb 4, 2014 at 9:25 PM, Guozhang Wang <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi Carl,
> >>
> >> For each partition the follower will also fetch data from the leader
> >> replica, even if there is no new data in the leader replicas.
> >>
> >> One thing you can try to increase replica.fetch.wait.max.ms (default
> value
> >> 500ms) so that the followers's fetching request frequency to the leader
> can
> >> be reduced, and see if that has some effect on the traffic.
> >>
> >> Guozhang
> >>
> >>
> >> On Tue, Feb 4, 2014 at 8:46 PM, Carl Lerche <[EMAIL PROTECTED]> wrote:
> >>
> >> > Hello,
> >> >
> >> > I'm running a 0.8.0 Kafka cluster of 3 servers. The service that it is
> >> > for is not in full production yet, so the data written to cluster is
> >> > minimal (seems to average between 100kb/s -> 300kb/s per server). I
> >> > have configured Kafka to have a 3 replicas. I am noticing that each
> >> > Kafka server is talking to all the others at a data rate of 40MB/s for
> >> > each server (so, a total of 80MB/s for each server). This
> >> > communication is constant.
> >> >
> >> > Is this normal? This seems like very strange behavior and I'm not
> >> > exactly sure how to debug.
> >> >
> >> > Thanks,
> >> > Carl
> >> >
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
>