Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> message loss


So looks like there is a jespen post coming on kafka 0.8 replication, based
on this thats circulating on twitter. https://www.refheap.com/17932/raw

Understanding that kafka isnt designed particularly to be partition
tolerant, the result is not completely surprising.

But my question is, is there something that can be done about the lost
messages?

From my understanding when broker n1 comes back on line, currently what
will happen is that the messages that were only on n1 will be
truncated/tossed while n1 is coming back to ISR. Please correct me if this
is not accurate.

Would it instead be possible to do something else with them, like sending
them to an internal lost messages topic, or log file where some manual
intervenion could be done on them, or a configuration property like
replay.truncated.messages=true could be set where the broker would send the
lost messages back onto the topic after ISR?

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB