Sure, and the disk could go bad, the machine itself could fail.

My point is that my experience of Kafka 0.72 has been that it is very reliable. The only time I have seen it go down is when the disk underneath fills up. So if one is going to write all the code to stream to disk *efficiently* from in-process, one should trade off that cost versus connecting to a process over localhost, which has been shown to do a very good job of just that.

It's fair to ask what if the broker process goes down. But it's fair to ask what if there is a bug in the stream-to-disk code you write? What if your process goes down?

I am not saying I am right. Just that engineering is about trade-offs, and a Kafka instance running right there on the same machine, might provide the reliability required. But it might not. As always, YMMV.

Philip
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB