Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa >> mail # dev >> Re: [jira] [Commented] (CHUKWA-581) Demux  do not support user custom reducer package name.


Copy link to this message
-
Re: [jira] [Commented] (CHUKWA-581) Demux  do not support user custom reducer package name.
Yes,this is better. I can fix the patch to apply this config.
--
Best regards,

Ivy Tang

On 2013-7-21, at 上午1:02, Eric Yang (JIRA) <[EMAIL PROTECTED]> wrote:

>
>    [ https://issues.apache.org/jira/browse/CHUKWA-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714480#comment-13714480 ]
>
> Eric Yang commented on CHUKWA-581:
> ----------------------------------
>
> How about make it more backward compatible with some tweaks to config?
>
> {code}
> <property>
>  <name>DataType1</name>
>  <value>mapperClass,reducerClass</value>
> </property>
>
> <property>
>  <name>mapperType1</name>
>  <value>mapperClass</value>
> </property>
>
> <property>
>  <name>ReduceType1</name>
>  <value>,reducerClass</value>
> </property>
> {code}
>
>> Demux  do not support user custom reducer package name.
>> -------------------------------------------------------
>>
>>                Key: CHUKWA-581
>>                URL: https://issues.apache.org/jira/browse/CHUKWA-581
>>            Project: Chukwa
>>         Issue Type: Improvement
>>         Components: MR Data Processors
>>   Affects Versions: 0.4.0, 0.5.0, 0.6.0
>>        Environment: Ubuntu 10.0.4,Hadoop 0.20
>>           Reporter: Yingzhong Xu
>>           Assignee: IvyTang
>>           Priority: Critical
>>             Labels: patch
>>            Fix For: 0.5.0, 0.6.0
>>
>>        Attachments: custom_reducer_conf.patch, custom_reducer.patch
>>
>>  Original Estimate: 24h
>> Remaining Estimate: 24h
>>
>> In ReduceProcessorFactory.java,
>> it only gets {reduceType}.class from this package,as following:
>> bq. String path = "org.apache.hadoop.chukwa.extraction.demux.processor.reducer." + reduceType;
>> So our custom reducer could only be placed in that package.
>> Thus, When we add our {demux}.jar to Hadoop classpath, since the Hadoop framework will not be able to merge the class files from multiple jar files with the same package name, custom reduce class can not be found.
>> I get a way to overcome this defect.Put custom reducer into a package named with "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+{custom}
>> Here,{custom} is a part of package name.Then in parser,we can write like this:
>> bq. Key.setReducerType("{custom}.YourReducer")
>> In this way, Hadoop framework could be able to find our reduce class in this partial user defined new package.
>> But I think it will be better to fix chukwa to remove the default package name and fully support custom package name for reducer.
>> Thanks for Eric Yang's help.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB