Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - [jira] [Commented] (KAFKA-713) Update Hadoop producer for Kafka 0.8 changes


Copy link to this message
-
[jira] [Commented] (KAFKA-713) Update Hadoop producer for Kafka 0.8 changes
"Neha Narkhede 2013-01-23, 18:45

    [ https://issues.apache.org/jira/browse/KAFKA-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560929#comment-13560929 ]

Neha Narkhede commented on KAFKA-713:
-------------------------------------

Thanks for the rebased patch, Sam. A few minor review suggestions -

1. KafkaOutputFormat

This will be a good time to add a few useful defaults to the Hadoop producer -
1.1 Remove max.message.size since it is obsolete in 0.8 as message size checks moved to the server
1.2 buffer.size is now send.buffer.bytes
1.3 Add compression.codec (0 = no compression, 1 = GZIP, 2 = Snappy). The Kafka producer defaults to no compression, but for Hadoop->Kafka pushes, compression will be useful to have. Should probably default to gzip/snappy

2. README
    REGISTER zkclient-20120522.jar; is no longer required

                
> Update Hadoop producer for Kafka 0.8 changes
> --------------------------------------------
>
>                 Key: KAFKA-713
>                 URL: https://issues.apache.org/jira/browse/KAFKA-713
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>            Reporter: Sam Shah
>            Assignee: Sam Shah
>              Labels: hadoop
>             Fix For: 0.8.1
>
>         Attachments: KAFKA-713.patch
>
>
> With the changes in Kafka 0.8, the Hadoop producer (in contrib) is busted due to changes in the way KeyedMessages are now handled. I will fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira