Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # dev >> About CHUKWA-581,Demux do not support user custom reducer package name.


Copy link to this message
-
Re: About CHUKWA-581,Demux do not support user custom reducer package name.
Hi,Eric
In order to applying to all multiple datatypes in Demux,init the Reducer class depends on the ChukwaRecordKey's datatype . I need the datatype in ChukwaRecordKey.

The ChukwaRecordKey needs to add a  field named dataType , showing this ChukwaRecord's datatype . And also i need add an argument in
parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter)

An argument String dataType needs to be passed to this method to build the ChukwaRecordKey.

Adding a field in ChukwaRecordKey ,and add an argument in method AbstractProcessor.parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter)

Is that acceptable? This is the only way i can think of to make the custom reducer more configurable.

--
Best regards,

Ivy Tang

On 2013-7-10, at 下午12:01, Eric Yang <[EMAIL PROTECTED]> wrote:

> Yes that will help.  Please include test case as well.  Thanks Ivy.
>
> regards,
> Eric
>
>
> On Mon, Jul 8, 2013 at 10:42 PM, IvyTang <[EMAIL PROTECTED]> wrote:
>
>> This issue has been reported
>> https://issues.apache.org/jira/browse/CHUKWA-581.
>>
>> And i think i can submit a patch later .
>>
>> Should i open a new jira  or just assign this CHUKWA-581 to myself?
>>
>> This can implement reducer outside chukwa.
>>
>> --
>> Best regards,
>>
>> Ivy Tang
>>
>> On 2013-7-9, at 下午1:22, Eric Yang <[EMAIL PROTECTED]> wrote:
>>
>>> The recommended change can improve usability to implement parsers outside
>>> of Chukwa packages.  +1 on the change.  Do you mind open a jira for this?
>>> Thanks
>>>
>>> regards,
>>> Eric
>>>
>>>
>>> On Mon, Jul 8, 2013 at 6:54 AM, 唐 颖 <[EMAIL PROTECTED]> wrote:
>>>
>>>> We can write our own mapper ,because MapProcessorFactory uses
>>>> Class.forName(parserClass).getConstructor().newInstance();
>>>> since we config the parserClass in config file.
>>>>
>>>> But the ReduceProcessorFactory inits  the reducer class using the
>>>> classname
>> "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+
>>>> reduceType.
>>>> So the reducer class must be put in package
>>>> org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named
>>>> reduceType.
>>>>
>>>> Why not we put the reducer  class name in config file either?
>>>> For exam, the {customdatatype_mapper} means mapper class
>>>> ,{customdatatype_reducer} means reducer class.
>>>>
>>>> This can give chukka more flexibility.
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Ivy Tang
>>>>
>>>>
>>
>>

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB