I'm developing a client for kafka 0.8. It looks like a fetch response will sometimes end with a partial message. I understand why this might be the case but it was unexpected and as far as I can tell undocumented.
Is my understanding correct or am i missing something?
Yeah, that is true. I thought I documented that, but looking at the protocol docs, it looks like I didn't.
I agree this is kind of a pain in the ass. It was an important optimization in 0.7 because we didn't know where the message boundaries were but in 0.8 we have a fast way to compute message boundaries and in fact we normally don't give out partial messages, I think this happens when you hit the size threshold of your fetch request (e.g. 1MB) instead of searching for the nearest message boundary we give you that chunk of log. I think we should consider just fixing it entirely in the next release--the perf hit is pretty minor and it is an annoyance and source of bugs for clients.
For now you have to handle it, so I added documentation to the protocol wiki.
On Wed, Jun 26, 2013 at 8:59 AM, Bob Potter <[EMAIL PROTECTED]> wrote:
This should happen when there is a backlog of data larger than the fetch size the consumer is using.
Also, just to be clear this problem is something client implementation needs to handle but not something the user of the client needs to handle.
On Thu, Jun 27, 2013 at 12:29 PM, Vadim Keylis <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext