Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Re: [jira] [Updated] (KAFKA-158) go consumer & producer to support compression


Copy link to this message
-
Re: [jira] [Updated] (KAFKA-158) go consumer & producer to support compression
Neha or others,

Do you see the problem in this stream?

thanks,
-jeff
On Thu, Oct 20, 2011 at 12:01 PM, Jeffrey Damick (Updated) (JIRA) <
[EMAIL PROTECTED]> wrote:

>
>     [
> https://issues.apache.org/jira/browse/KAFKA-158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Jeffrey Damick updated KAFKA-158:
> ---------------------------------
>
>    Attachment: kafka_158_go_compress.patch
>
> patch to add ability to compressed messages
>
> Seeing strange behavior from kafka when sending the compressed messages:
>
> go client sends this:
>
> 00 00 00 36 00 00 00 04 74 65 73 74 00 00 00 00 00 00 00 26 00 00 00 22 01
> 01 F0 17 43 A5 1F 8B 08 00 00 00 00 00 04 FF 4A CE CF 2D 00 04 00 00 FF FF
> 3A 6F 0A CB 04 00 00 00
>
> and servers gives:
>
> [2011-10-20 11:51:50,106] DEBUG Listening to new connection from /
> 127.0.0.1:59861 (kafka.network.Processor)
> [2011-10-20 11:51:50,107] TRACE 54 bytes read from /127.0.0.1:59861(kafka.network.Processor)
> [2011-10-20 11:51:50,115] TRACE Handling produce request from /
> 127.0.0.1:59861 (kafka.request.logger)
> [2011-10-20 11:51:50,119] TRACE Producer request ProducerRequest(test,0,38)
> (kafka.request.logger)
> [2011-10-20 11:51:50,119] DEBUG makeNext() in deepIterator: innerDone > true (kafka.message.ByteBufferMessageSet)
> [2011-10-20 11:51:50,119] TRACE Remaining bytes in iterator = 34
> (kafka.message.ByteBufferMessageSet)
> [2011-10-20 11:51:50,119] TRACE size of data = 34
> (kafka.message.ByteBufferMessageSet)
> [2011-10-20 11:51:50,120] DEBUG Message is compressed. Valid byte count = 0
> (kafka.message.ByteBufferMessageSet)
> [2011-10-20 11:51:50,133] DEBUG makeNext() in deepIterator: innerDone > true (kafka.message.ByteBufferMessageSet)
> [2011-10-20 11:51:50,133] TRACE Remaining bytes in iterator = 0
> (kafka.message.ByteBufferMessageSet)
> [2011-10-20 11:51:50,133] TRACE size of data = 1668246896
> (kafka.message.ByteBufferMessageSet)
> [2011-10-20 11:51:50,134] ERROR Error processing ProduceRequest on test:0
> (kafka.server.KafkaRequestHandlers)
> kafka.common.InvalidMessageSizeException: invalid message size: 1668246896
> only received bytes: 0 at 0( possible causes (1) a single message larger
> than the fetch size; (2) log corruption )
>
>
>
> > go consumer & producer to support compression
> > ---------------------------------------------
> >
> >                 Key: KAFKA-158
> >                 URL: https://issues.apache.org/jira/browse/KAFKA-158
> >             Project: Kafka
> >          Issue Type: Bug
> >          Components: clients
> >    Affects Versions: 0.7
> >            Reporter: Jeffrey Damick
> >            Priority: Minor
> >              Labels: go-client
> >         Attachments: KAFKA-158-try2.patch, KAFKA-158.patch,
> kafka_158_go_compress.patch
> >
> >
> > As related to KAFKA-79, the go consumer and producer needs to support the
> compression attribute per
> https://cwiki.apache.org/confluence/display/KAFKA/Compression.
> > Can someone assign this to me, i'll add support and create a patch.
> > thanks
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators:
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB