Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> Troubles with compressed message set


Copy link to this message
-
Troubles with compressed message set
I'm working on a client and I'm running into difficulties with
compressed message sets. I am able to produce them fine, but when I go
to fetch them, things seem strange.

I am sending a message who's value is a compressed message set. The
inner message set contains a single message. Specifically what looks
weird is that the key of the top message looks corrupt. Here is a trace
of my payloads:

https://gist.github.com/bf134906f6559b0f54ad

See the "???" down in the FetchResponse for what I mean. Also the magic
byte and attributes are wrong

The data in the Kafka log for this partition matches what I get back for
the MessageSet in the FetchResponse:

\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00;\xf5#\xc2N\x00\x01\xff\xff\xff\xff\x00\x00\x00-\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00c`\x80\x03\x89P\xf7\xef\xccL
\x16sZ~>\x90bw\x8f\xf2\x0c\x08HM\x01\x00\xc5\x93\xd3<$\x00\x00\x00

Another bit of weirdness here is how MessageSets are encoded. Everywhere
else in the API, we prefix a repeated element with a size of int32. When
encoding MessageSets, if I follow this convention, Kafka rejects the
produce request - if I exclude that int32 it works fine. I don't know if
this was intentional or not, but it is somewhat annoying and
inconsistent. When decoding MessageSets, I have to do-while instead of
iterate a known number of times.

Cheers
-David

 
+
Jay Kreps 2013-01-30, 16:19
+
David Arthur 2013-01-30, 21:00
+
Jay Kreps 2013-01-30, 21:17
+
David Arthur 2013-01-30, 21:23
+
David Arthur 2013-01-30, 21:07
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB