Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa >> mail # dev >> About CHUKWA-581,Demux do not support user custom reducer package name.


+
唐 颖 2013-07-08, 13:54
+
Eric Yang 2013-07-09, 05:22
+
IvyTang 2013-07-09, 05:42
+
Eric Yang 2013-07-10, 04:01
+
IvyTang 2013-07-12, 07:04
+
IvyTang 2013-07-12, 09:01
Copy link to this message
-
Re: About CHUKWA-581,Demux do not support user custom reducer package name.
Yes, Chunk.getDataType() is the right way to do it.  Thanks

regards,
Eric
On Fri, Jul 12, 2013 at 2:01 AM, IvyTang <[EMAIL PROTECTED]> wrote:

> Fix:
>   Don't need to add the argument dataType in
> AbstractProcessor.parse(String recordEntry,
> OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter).
>   I can get the datatype  from the chunk.
> --
> Best regards,
>
> Ivy Tang
>
> On 2013-7-12, at 下午3:04, IvyTang <[EMAIL PROTECTED]> wrote:
>
> > Hi,Eric
> > In order to applying to all multiple datatypes in Demux,init the Reducer
> class depends on the ChukwaRecordKey's datatype . I need the datatype in
> ChukwaRecordKey.
> >
> > The ChukwaRecordKey needs to add a  field named dataType , showing this
> ChukwaRecord's datatype . And also i need add an argument in
> > parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord>
> output, Reporter reporter)
> >
> > An argument String dataType needs to be passed to this method to build
> the ChukwaRecordKey.
> >
> > Adding a field in ChukwaRecordKey ,and add an argument in method
> AbstractProcessor.parse(String recordEntry,
> OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter)
> >
> > Is that acceptable? This is the only way i can think of to make the
> custom reducer more configurable.
> >
> > --
> > Best regards,
> >
> > Ivy Tang
> >
> > On 2013-7-10, at 下午12:01, Eric Yang <[EMAIL PROTECTED]> wrote:
> >
> >> Yes that will help.  Please include test case as well.  Thanks Ivy.
> >>
> >> regards,
> >> Eric
> >>
> >>
> >> On Mon, Jul 8, 2013 at 10:42 PM, IvyTang <[EMAIL PROTECTED]> wrote:
> >>
> >>> This issue has been reported
> >>> https://issues.apache.org/jira/browse/CHUKWA-581.
> >>>
> >>> And i think i can submit a patch later .
> >>>
> >>> Should i open a new jira  or just assign this CHUKWA-581 to myself?
> >>>
> >>> This can implement reducer outside chukwa.
> >>>
> >>> --
> >>> Best regards,
> >>>
> >>> Ivy Tang
> >>>
> >>> On 2013-7-9, at 下午1:22, Eric Yang <[EMAIL PROTECTED]> wrote:
> >>>
> >>>> The recommended change can improve usability to implement parsers
> outside
> >>>> of Chukwa packages.  +1 on the change.  Do you mind open a jira for
> this?
> >>>> Thanks
> >>>>
> >>>> regards,
> >>>> Eric
> >>>>
> >>>>
> >>>> On Mon, Jul 8, 2013 at 6:54 AM, 唐 颖 <[EMAIL PROTECTED]> wrote:
> >>>>
> >>>>> We can write our own mapper ,because MapProcessorFactory uses
> >>>>> Class.forName(parserClass).getConstructor().newInstance();
> >>>>> since we config the parserClass in config file.
> >>>>>
> >>>>> But the ReduceProcessorFactory inits  the reducer class using the
> >>>>> classname
> >>> "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+
> >>>>> reduceType.
> >>>>> So the reducer class must be put in package
> >>>>> org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named
> >>>>> reduceType.
> >>>>>
> >>>>> Why not we put the reducer  class name in config file either?
> >>>>> For exam, the {customdatatype_mapper} means mapper class
> >>>>> ,{customdatatype_reducer} means reducer class.
> >>>>>
> >>>>> This can give chukka more flexibility.
> >>>>>
> >>>>> --
> >>>>> Best regards,
> >>>>>
> >>>>> Ivy Tang
> >>>>>
> >>>>>
> >>>
> >>>
> >
>
>