Thanks very much for the reply Neha! So I swapped out the consumer that
processes the messages with one that just prints them. It does indeed
achieve a much better rate at peaks but can still nearly zero out (if not
completely zero out). I plotted the messages printed in graphite to show
the behaviour I'm seeing (this is messages printed per second):

The peaks are over ten thousand per second and the troughs can go below 10
per second just prior to another peak. I know that there are plenty of
messages available because the ones currently being processed are still
from Friday afternoon, so this may or may not have something to do with
this pattern.

Is there anything I can do to avoid the periods of lower performance?
Ideally I would be processing messages as soon as they are written.
On Sun, Apr 21, 2013 at 8:49 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB