Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # dev - Re: [jira] [Updated] (KAFKA-158) go consumer & producer to support compression


Copy link to this message
-
Re: [jira] [Updated] (KAFKA-158) go consumer & producer to support compression
Jeffrey Damick 2011-10-20, 16:04
Neha or others,

Do you see the problem in this stream?

thanks,
-jeff
On Thu, Oct 20, 2011 at 12:01 PM, Jeffrey Damick (Updated) (JIRA) <
[EMAIL PROTECTED]> wrote:

>
>     [
> https://issues.apache.org/jira/browse/KAFKA-158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Jeffrey Damick updated KAFKA-158:
> ---------------------------------
>
>    Attachment: kafka_158_go_compress.patch
>
> patch to add ability to compressed messages
>
> Seeing strange behavior from kafka when sending the compressed messages:
>
> go client sends this:
>
> 00 00 00 36 00 00 00 04 74 65 73 74 00 00 00 00 00 00 00 26 00 00 00 22 01
> 01 F0 17 43 A5 1F 8B 08 00 00 00 00 00 04 FF 4A CE CF 2D 00 04 00 00 FF FF
> 3A 6F 0A CB 04 00 00 00
>
> and servers gives:
>
> [2011-10-20 11:51:50,106] DEBUG Listening to new connection from /
> 127.0.0.1:59861 (kafka.network.Processor)
> [2011-10-20 11:51:50,107] TRACE 54 bytes read from /127.0.0.1:59861(kafka.network.Processor)
> [2011-10-20 11:51:50,115] TRACE Handling produce request from /
> 127.0.0.1:59861 (kafka.request.logger)
> [2011-10-20 11:51:50,119] TRACE Producer request ProducerRequest(test,0,38)
> (kafka.request.logger)
> [2011-10-20 11:51:50,119] DEBUG makeNext() in deepIterator: innerDone > true (kafka.message.ByteBufferMessageSet)
> [2011-10-20 11:51:50,119] TRACE Remaining bytes in iterator = 34
> (kafka.message.ByteBufferMessageSet)
> [2011-10-20 11:51:50,119] TRACE size of data = 34
> (kafka.message.ByteBufferMessageSet)
> [2011-10-20 11:51:50,120] DEBUG Message is compressed. Valid byte count = 0
> (kafka.message.ByteBufferMessageSet)
> [2011-10-20 11:51:50,133] DEBUG makeNext() in deepIterator: innerDone > true (kafka.message.ByteBufferMessageSet)
> [2011-10-20 11:51:50,133] TRACE Remaining bytes in iterator = 0
> (kafka.message.ByteBufferMessageSet)
> [2011-10-20 11:51:50,133] TRACE size of data = 1668246896
> (kafka.message.ByteBufferMessageSet)
> [2011-10-20 11:51:50,134] ERROR Error processing ProduceRequest on test:0
> (kafka.server.KafkaRequestHandlers)
> kafka.common.InvalidMessageSizeException: invalid message size: 1668246896
> only received bytes: 0 at 0( possible causes (1) a single message larger
> than the fetch size; (2) log corruption )
>
>
>
> > go consumer & producer to support compression
> > ---------------------------------------------
> >
> >                 Key: KAFKA-158
> >                 URL: https://issues.apache.org/jira/browse/KAFKA-158
> >             Project: Kafka
> >          Issue Type: Bug
> >          Components: clients
> >    Affects Versions: 0.7
> >            Reporter: Jeffrey Damick
> >            Priority: Minor
> >              Labels: go-client
> >         Attachments: KAFKA-158-try2.patch, KAFKA-158.patch,
> kafka_158_go_compress.patch
> >
> >
> > As related to KAFKA-79, the go consumer and producer needs to support the
> compression attribute per
> https://cwiki.apache.org/confluence/display/KAFKA/Compression.
> > Can someone assign this to me, i'll add support and create a patch.
> > thanks
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators:
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>