Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa >> mail # dev >> About CHUKWA-581,Demux do not support user custom reducer package name.


+
唐 颖 2013-07-08, 13:54
+
Eric Yang 2013-07-09, 05:22
+
IvyTang 2013-07-09, 05:42
+
Eric Yang 2013-07-10, 04:01
+
IvyTang 2013-07-12, 07:04
+
IvyTang 2013-07-12, 09:01
Copy link to this message
-
Re: About CHUKWA-581,Demux do not support user custom reducer package name.
Yes, Chunk.getDataType() is the right way to do it.  Thanks

regards,
Eric
On Fri, Jul 12, 2013 at 2:01 AM, IvyTang <[EMAIL PROTECTED]> wrote:

> Fix:
>   Don't need to add the argument dataType in
> AbstractProcessor.parse(String recordEntry,
> OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter).
>   I can get the datatype  from the chunk.
> --
> Best regards,
>
> Ivy Tang
>
> On 2013-7-12, at 下午3:04, IvyTang <[EMAIL PROTECTED]> wrote:
>
> > Hi,Eric
> > In order to applying to all multiple datatypes in Demux,init the Reducer
> class depends on the ChukwaRecordKey's datatype . I need the datatype in
> ChukwaRecordKey.
> >
> > The ChukwaRecordKey needs to add a  field named dataType , showing this
> ChukwaRecord's datatype . And also i need add an argument in
> > parse(String recordEntry, OutputCollector<ChukwaRecordKey, ChukwaRecord>
> output, Reporter reporter)
> >
> > An argument String dataType needs to be passed to this method to build
> the ChukwaRecordKey.
> >
> > Adding a field in ChukwaRecordKey ,and add an argument in method
> AbstractProcessor.parse(String recordEntry,
> OutputCollector<ChukwaRecordKey, ChukwaRecord> output, Reporter reporter)
> >
> > Is that acceptable? This is the only way i can think of to make the
> custom reducer more configurable.
> >
> > --
> > Best regards,
> >
> > Ivy Tang
> >
> > On 2013-7-10, at 下午12:01, Eric Yang <[EMAIL PROTECTED]> wrote:
> >
> >> Yes that will help.  Please include test case as well.  Thanks Ivy.
> >>
> >> regards,
> >> Eric
> >>
> >>
> >> On Mon, Jul 8, 2013 at 10:42 PM, IvyTang <[EMAIL PROTECTED]> wrote:
> >>
> >>> This issue has been reported
> >>> https://issues.apache.org/jira/browse/CHUKWA-581.
> >>>
> >>> And i think i can submit a patch later .
> >>>
> >>> Should i open a new jira  or just assign this CHUKWA-581 to myself?
> >>>
> >>> This can implement reducer outside chukwa.
> >>>
> >>> --
> >>> Best regards,
> >>>
> >>> Ivy Tang
> >>>
> >>> On 2013-7-9, at 下午1:22, Eric Yang <[EMAIL PROTECTED]> wrote:
> >>>
> >>>> The recommended change can improve usability to implement parsers
> outside
> >>>> of Chukwa packages.  +1 on the change.  Do you mind open a jira for
> this?
> >>>> Thanks
> >>>>
> >>>> regards,
> >>>> Eric
> >>>>
> >>>>
> >>>> On Mon, Jul 8, 2013 at 6:54 AM, 唐 颖 <[EMAIL PROTECTED]> wrote:
> >>>>
> >>>>> We can write our own mapper ,because MapProcessorFactory uses
> >>>>> Class.forName(parserClass).getConstructor().newInstance();
> >>>>> since we config the parserClass in config file.
> >>>>>
> >>>>> But the ReduceProcessorFactory inits  the reducer class using the
> >>>>> classname
> >>> "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+
> >>>>> reduceType.
> >>>>> So the reducer class must be put in package
> >>>>> org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named
> >>>>> reduceType.
> >>>>>
> >>>>> Why not we put the reducer  class name in config file either?
> >>>>> For exam, the {customdatatype_mapper} means mapper class
> >>>>> ,{customdatatype_reducer} means reducer class.
> >>>>>
> >>>>> This can give chukka more flexibility.
> >>>>>
> >>>>> --
> >>>>> Best regards,
> >>>>>
> >>>>> Ivy Tang
> >>>>>
> >>>>>
> >>>
> >>>
> >
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB