Re: kafka producer - retry messages
Sure, and the disk could go bad, the machine itself could fail.
My point is that my experience of Kafka 0.72 has been that it is very reliable. The only time I have seen it go down is when the disk underneath fills up. So if one is going to write all the code to stream to disk *efficiently* from in-process, one should trade off that cost versus connecting to a process over localhost, which has been shown to do a very good job of just that.
It's fair to ask what if the broker process goes down. But it's fair to ask what if there is a bug in the stream-to-disk code you write? What if your process goes down?
I am not saying I am right. Just that engineering is about trade-offs, and a Kafka instance running right there on the same machine, might provide the reliability required. But it might not. As always, YMMV.