Hmm it is highly unlikely that that is the culprit... There is lots of
bandwidth available for me to use. I will definitely keep that in mind
though. I was working on this today and have some tidbits of additional
information and thoughts that you might be able to shed some light on:

   - I mentioned I have 2 consumers, but each consumer is running with 8
   threads for this topic (and each consumer has 8 cores available).
   - When I initially asked for help the brokers were configured with
   num.partitions=1, I've since tried higher numbers (3, 64) and haven't seen
   much of an improvement aside from forcing both consumer apps to handle
   messages (with the overall performance not changing much).
   - I ran into this article
tried a variety of options of queuedchunks.max and fetch.size with no
   significant results (simply meaning it did not achieve the goal of me
   constantly processing hundreds or thousands of messages per second, which
   is similar to the rate of input). I would not be surprised if I'm wrong but
   this made me start to think that the problem may lie outside of the
   - Would the combination of a high number of partitions (64) and a high
   log.flush.interval (10k) prevent logs from flushing as often as they need
   to for my desired rate of consumption (even with

Despite the changes I mentioned the behaviour is still the consumers
receiving larger spikes of messages mixed with periods of complete
inactivity and overall a long delay between messages being written and
messages being read (about 2 minutes). Anyway... as always I greatly
appreciate any help.

On Sun, Apr 21, 2013 at 8:50 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB