Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa >> mail # dev >> About CHUKWA-581,Demux do not support user custom reducer package name.


+
唐 颖 2013-07-08, 13:54
+
Eric Yang 2013-07-09, 05:22
+
IvyTang 2013-07-09, 05:42
+
Eric Yang 2013-07-10, 04:01
+
IvyTang 2013-07-12, 07:04
Copy link to this message
-
Re: About CHUKWA-581,Demux do not support user custom reducer package name.
Fix:
  Don't need to add the argument dataType in AbstractProcessor.parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter).
  I can get the datatype  from the chunk.
--
Best regards,

Ivy Tang

On 2013-7-12, at 下午3:04, IvyTang <[EMAIL PROTECTED]> wrote:

> Hi,Eric
> In order to applying to all multiple datatypes in Demux,init the Reducer class depends on the ChukwaRecordKey's datatype . I need the datatype in ChukwaRecordKey.
>
> The ChukwaRecordKey needs to add a  field named dataType , showing this ChukwaRecord's datatype . And also i need add an argument in
> parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter)
>
> An argument String dataType needs to be passed to this method to build the ChukwaRecordKey.
>
> Adding a field in ChukwaRecordKey ,and add an argument in method AbstractProcessor.parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter)
>
> Is that acceptable? This is the only way i can think of to make the custom reducer more configurable.
>
> --
> Best regards,
>
> Ivy Tang
>
> On 2013-7-10, at 下午12:01, Eric Yang <[EMAIL PROTECTED]> wrote:
>
>> Yes that will help.  Please include test case as well.  Thanks Ivy.
>>
>> regards,
>> Eric
>>
>>
>> On Mon, Jul 8, 2013 at 10:42 PM, IvyTang <[EMAIL PROTECTED]> wrote:
>>
>>> This issue has been reported
>>> https://issues.apache.org/jira/browse/CHUKWA-581.
>>>
>>> And i think i can submit a patch later .
>>>
>>> Should i open a new jira  or just assign this CHUKWA-581 to myself?
>>>
>>> This can implement reducer outside chukwa.
>>>
>>> --
>>> Best regards,
>>>
>>> Ivy Tang
>>>
>>> On 2013-7-9, at 下午1:22, Eric Yang <[EMAIL PROTECTED]> wrote:
>>>
>>>> The recommended change can improve usability to implement parsers outside
>>>> of Chukwa packages.  +1 on the change.  Do you mind open a jira for this?
>>>> Thanks
>>>>
>>>> regards,
>>>> Eric
>>>>
>>>>
>>>> On Mon, Jul 8, 2013 at 6:54 AM, 唐 颖 <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> We can write our own mapper ,because MapProcessorFactory uses
>>>>> Class.forName(parserClass).getConstructor().newInstance();
>>>>> since we config the parserClass in config file.
>>>>>
>>>>> But the ReduceProcessorFactory inits  the reducer class using the
>>>>> classname
>>> "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+
>>>>> reduceType.
>>>>> So the reducer class must be put in package
>>>>> org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named
>>>>> reduceType.
>>>>>
>>>>> Why not we put the reducer  class name in config file either?
>>>>> For exam, the {customdatatype_mapper} means mapper class
>>>>> ,{customdatatype_reducer} means reducer class.
>>>>>
>>>>> This can give chukka more flexibility.
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> Ivy Tang
>>>>>
>>>>>
>>>
>>>
>

+
Eric Yang 2013-07-13, 20:35