Carl, looking at the boundary chart it looks like you have periods of no
traffic also... prior to the spikes.

I also noticed you are using AWS from your logs, what instance types are
you using?  Do you have any network checks in place?

The logs show underReplication=true which leads towards what Joel was
theorizing as the issue.

Do you track stats on the cluster?
http://kafka.apache.org/documentation.html#monitoring I would expect
correlation of changes in the kafka stats and the boundary chart.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Fri, Feb 7, 2014 at 2:47 AM, Carl Lerche <[EMAIL PROTECTED]> wrote:
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB