Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> Troubles with compressed message set


Copy link to this message
-
Troubles with compressed message set
I'm working on a client and I'm running into difficulties with
compressed message sets. I am able to produce them fine, but when I go
to fetch them, things seem strange.

I am sending a message who's value is a compressed message set. The
inner message set contains a single message. Specifically what looks
weird is that the key of the top message looks corrupt. Here is a trace
of my payloads:

https://gist.github.com/bf134906f6559b0f54ad

See the "???" down in the FetchResponse for what I mean. Also the magic
byte and attributes are wrong

The data in the Kafka log for this partition matches what I get back for
the MessageSet in the FetchResponse:

\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00;\xf5#\xc2N\x00\x01\xff\xff\xff\xff\x00\x00\x00-\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00c`\x80\x03\x89P\xf7\xef\xccL
\x16sZ~>\x90bw\x8f\xf2\x0c\x08HM\x01\x00\xc5\x93\xd3<$\x00\x00\x00

Another bit of weirdness here is how MessageSets are encoded. Everywhere
else in the API, we prefix a repeated element with a size of int32. When
encoding MessageSets, if I follow this convention, Kafka rejects the
produce request - if I exclude that int32 it works fine. I don't know if
this was intentional or not, but it is somewhat annoying and
inconsistent. When decoding MessageSets, I have to do-while instead of
iterate a known number of times.

Cheers
-David

 
+
Jay Kreps 2013-01-30, 16:19
+
David Arthur 2013-01-30, 21:00
+
Jay Kreps 2013-01-30, 21:17
+
David Arthur 2013-01-30, 21:23
+
David Arthur 2013-01-30, 21:07