Home | About | Sematext search-lucene.com search-hadoop.com search-devops.com metrics + logs = try SPM and Logsene for free
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> CallbackHandler in Kafka 0.8


Copy link to this message
-
Re: CallbackHandler in Kafka 0.8
Hi Nitin,

However, this depends on your event rate - i.e., if it's a very high
event rate then the batch threshold is reached and send happens before
that time interval. Furthermore, even if event rate is low, the
de-duplication applies to a time-window of at most
queue.buffering.max.ms).

In 0.7, batches would be taken and the call-back handler's
beforeSendingData method would be invoked on those batches. i.e., you
can achieve the same effect as you did in 0.7 by keeping keeping an
additional buffer (before the actual send) of
queue.buffering.max.messages (i.e., the batch size) and de-duping
within that buffer.

Thanks,

Joel

On Wed, Jul 3, 2013 at 1:33 AM, Nitin Supekar <[EMAIL PROTECTED]> wrote:

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB