David Arthur 2013-01-30, 15:27
MessageSets aren't size delimited because that format is shared with
producer/consumer and the kafka log itself. The log itself is just a big
sequence of messages and any subset that begins and ends at valid message
boundaries is a valid message set. This means that message sets are size
prefixed only as part of the request/response. Not sure if that is what you
It's hard to see the cause of the error you describe. I don't suppose you
could send me a snapshot of your client to reproduce locally?
On Wed, Jan 30, 2013 at 7:26 AM, David Arthur <[EMAIL PROTECTED]> wrote:
> I'm working on a client and I'm running into difficulties with compressed
> message sets. I am able to produce them fine, but when I go to fetch them,
> things seem strange.
> I am sending a message who's value is a compressed message set. The inner
> message set contains a single message. Specifically what looks weird is
> that the key of the top message looks corrupt. Here is a trace of my
> See the "???" down in the FetchResponse for what I mean. Also the magic
> byte and attributes are wrong
> The data in the Kafka log for this partition matches what I get back for
> the MessageSet in the FetchResponse:
> Another bit of weirdness here is how MessageSets are encoded. Everywhere
> else in the API, we prefix a repeated element with a size of int32. When
> encoding MessageSets, if I follow this convention, Kafka rejects the
> produce request - if I exclude that int32 it works fine. I don't know if
> this was intentional or not, but it is somewhat annoying and inconsistent.
> When decoding MessageSets, I have to do-while instead of iterate a known
> number of times.
David Arthur 2013-01-30, 21:00
Jay Kreps 2013-01-30, 21:17
David Arthur 2013-01-30, 21:23
David Arthur 2013-01-30, 21:07