Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Truncated messages in fetch response


+
Bob Potter 2013-06-26, 16:00
Copy link to this message
-
Re: Truncated messages in fetch response
Yeah, that is true. I thought I documented that, but looking at the
protocol docs, it looks like I didn't.

I agree this is kind of a pain in the ass. It was an important
optimization in 0.7 because we didn't know where the message
boundaries were but in 0.8 we have a fast way to compute message
boundaries and in fact we normally don't give out partial messages, I
think this happens when you hit the size threshold of your fetch
request (e.g. 1MB) instead of searching for the nearest message
boundary we give you that chunk of log. I think we should consider
just fixing it entirely in the next release--the perf hit is pretty
minor and it is an annoyance and source of bugs for clients.

For now you have to handle it, so I added documentation to the protocol wiki.

Sorry!

-Jay

On Wed, Jun 26, 2013 at 8:59 AM, Bob Potter <[EMAIL PROTECTED]> wrote:
> Howdy,
>
> I'm developing a client for kafka 0.8. It looks like a fetch response will
> sometimes end with a partial message. I understand why this might be the
> case but it was unexpected and as far as I can tell undocumented.
>
> Is my understanding correct or am i missing something?
>
> -Bob

 
+
Vadim Keylis 2013-06-27, 19:30
+
Bob Potter 2013-06-27, 22:30
+
Milind Parikh 2013-06-27, 22:53
+
Jay Kreps 2013-06-27, 23:06
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB