Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Truncated messages in fetch response


+
Bob Potter 2013-06-26, 16:00
Copy link to this message
-
Re: Truncated messages in fetch response
Jay Kreps 2013-06-26, 16:21
Yeah, that is true. I thought I documented that, but looking at the
protocol docs, it looks like I didn't.

I agree this is kind of a pain in the ass. It was an important
optimization in 0.7 because we didn't know where the message
boundaries were but in 0.8 we have a fast way to compute message
boundaries and in fact we normally don't give out partial messages, I
think this happens when you hit the size threshold of your fetch
request (e.g. 1MB) instead of searching for the nearest message
boundary we give you that chunk of log. I think we should consider
just fixing it entirely in the next release--the perf hit is pretty
minor and it is an annoyance and source of bugs for clients.

For now you have to handle it, so I added documentation to the protocol wiki.

Sorry!

-Jay

On Wed, Jun 26, 2013 at 8:59 AM, Bob Potter <[EMAIL PROTECTED]> wrote:
> Howdy,
>
> I'm developing a client for kafka 0.8. It looks like a fetch response will
> sometimes end with a partial message. I understand why this might be the
> case but it was unexpected and as far as I can tell undocumented.
>
> Is my understanding correct or am i missing something?
>
> -Bob

 
+
Vadim Keylis 2013-06-27, 19:30
+
Bob Potter 2013-06-27, 22:30
+
Milind Parikh 2013-06-27, 22:53
+
Jay Kreps 2013-06-27, 23:06