Hi, all I'm working on snappy compression in our Node.js client ( https://github.com/SOHU-Co/kafka-node),but I found the offset of message always 0 after snappy compression, for example: <Buffer 00 00 00 00 00 00 00 00 00 00 00 19 3e 69 5f c8 00 00 00 00 00 00 00 00 00 0b 7b 22 6f 6e 65 22 3a 22 31 22 7d>, I try to test it in kafka-cosole-producer.sh, but it seems not support snappy. So I can't figure out what's wrong with my client. Did anyone encounter the same problem? Thanks.
Currently the console producer only supports GZIP. Could you please file a JIRA for the snappy support? On Thu, Mar 27, 2014 at 5:21 AM, 小宇 <[EMAIL PROTECTED]> wrote:
Re: Kafka snappy compression
Magnus Edenhill 2014-03-27, 12:37
Some notes on this: - remember that Messages are wrapped in a compressed MessageSet that is wrapped in an outer Message which is wrapped in an outer MessageSet (the mind boggles). - the Kafka snappy compression has a non-standard framing - snappy-java (prefix "\x82SNAPPY\x0") - this framing is specific to the java implementation and is not in the official snappy spec or implementation.
You can use the rdkafka_example client from librdkafka to send snappy-compressed messages: ./rdkafka_example -P -t topic -p 0 -b mybroker -z snappy