Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Surprisingly high network traffic between kafka servers


Copy link to this message
-
Re: Surprisingly high network traffic between kafka servers
Can you enable DEBUG logging in log4j and see what requests are coming in?

-Jay
On Tue, Feb 4, 2014 at 9:51 PM, Carl Lerche <[EMAIL PROTECTED]> wrote:

> Hi Jay,
>
> I do not believe that I have changed the replica.fetch.wait.max.ms
> setting. Here I have included the kafka config as well as a snapshot
> of jnettop from one of the servers.
>
> https://gist.github.com/carllerche/4f2cf0f0f6d1e891f482
>
> The bottom row (89.9K/s) is the producer (it lives on a Kafka server).
> The top two rows are Kafkas on other servers, you can see the combined
> throughput is ~80MB/s
>
> On Tue, Feb 4, 2014 at 9:36 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> > No this is not normal.
> >
> > Checking twice a second (using 500ms default) for new data shouldn't
> cause
> > high network traffic (that should be like < 1KB of overhead). I don't
> think
> > that explains things. Is it possible that setting has been overridden?
> >
> > -Jay
> >
> >
> > On Tue, Feb 4, 2014 at 9:25 PM, Guozhang Wang <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi Carl,
> >>
> >> For each partition the follower will also fetch data from the leader
> >> replica, even if there is no new data in the leader replicas.
> >>
> >> One thing you can try to increase replica.fetch.wait.max.ms (default
> value
> >> 500ms) so that the followers's fetching request frequency to the leader
> can
> >> be reduced, and see if that has some effect on the traffic.
> >>
> >> Guozhang
> >>
> >>
> >> On Tue, Feb 4, 2014 at 8:46 PM, Carl Lerche <[EMAIL PROTECTED]> wrote:
> >>
> >> > Hello,
> >> >
> >> > I'm running a 0.8.0 Kafka cluster of 3 servers. The service that it is
> >> > for is not in full production yet, so the data written to cluster is
> >> > minimal (seems to average between 100kb/s -> 300kb/s per server). I
> >> > have configured Kafka to have a 3 replicas. I am noticing that each
> >> > Kafka server is talking to all the others at a data rate of 40MB/s for
> >> > each server (so, a total of 80MB/s for each server). This
> >> > communication is constant.
> >> >
> >> > Is this normal? This seems like very strange behavior and I'm not
> >> > exactly sure how to debug.
> >> >
> >> > Thanks,
> >> > Carl
> >> >
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB