Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa, mail # user - Changing default demuxer


Copy link to this message
-
Re: Changing default demuxer
Bill Graham 2010-09-30, 03:45
This is a bug in the documentation that I've noticed before. I've
opened a JIRA for it:
https://issues.apache.org/jira/browse/CHUKWA-528

The default processor is actually DefaultProcessor.

On Wed, Sep 29, 2010 at 5:43 AM, Stuti Awasthi
<[EMAIL PROTECTED]> wrote:
> Hi all,
>
>
>
> I have applied the patch and set the property in chukwa-demux-conf.xml file
> as follows :
>
> <property>
>
>  <name>chukwa.demux.mapper.default.processor</name>
>
>  <value>com.test.genericdemuxer.xyz</value>
>
> </property>
>
>
>
> <property>
>
>  <name>chukwa.demux.reducer.default.processor</name>
>
>  <value>com.test.genericdemuxer.xyz</value>
>
> </property>
>
> I tested the whole process from log collection to creation of .evt files in
> repos, that worked fine.
>
> Please note I have not added any datatype for my logtype in demux-conf file
> so that it can go to my processor that I have set in the property file.
>
>
>
> I also tested the whole process again without these entries in the
> demux-conf file to check the default behavior of chukwa, but surprisingly
> that also worked fine without any error.
>
>
>
> There was a difference in the sizes of the .evt file that generated from
> both the process. I am not able to figure out that whether my processor is
> getting used or not.
>
>
>
> How can I get to know that it is using my processor?
>
>
>
> I also have a query that in demux.java , we have code where
>  DefaultProcessor is hardcoded, and in documentation it is written that
> TSProcessor will the default processor will be used if we do not specify the
> datatype for particular log type in demux-con.xml file.
>
> String defaultProcessor = Demux.jobConf.get(
>
>             "chukwa.demux.mapper.default.processor",
>
>
> "org.apache.hadoop.chukwa.extraction.demux.processor.mapper.DefaultProcessor");
>
>
>
> String processorClass = Demux.jobConf.get(chunk.getDataType(),
>
>                 defaultProcessor);
>
>
>
> I think that either I am missing something or not able to clearly understand
> the flow.
>
> Please suggest.
>
>
>
> Regards
>
> Stuti Awasthi
>
>
>
> -----Original Message-----
>
> From: Bill Graham [mailto:[EMAIL PROTECTED]]
>
> Sent: Tuesday, September 28, 2010 9:32 AM
>
> To: [EMAIL PROTECTED]
>
> Cc: Jaydeep Ayachit
>
> Subject: Re: Changing default demuxer
>
>
>
> This can be changed with the following configs:
>
>
>
> chukwa.demux.mapper.default.processor
>
> chukwa.demux.reducer.default.processor
>
>
>
> Reference JIRA:
>
> https://issues.apache.org/jira/browse/CHUKWA-473
>
>
>
> This is part from Chukwa 0.4.0.
>
>
>
>
>
> On Mon, Sep 27, 2010 at 5:47 AM, Jaydeep Ayachit
>
> <[EMAIL PROTECTED]> wrote:
>
>> Hello,
>
>>
>
>>
>
>>
>
>> Chukwa by default uses TSProcessor demuxer for data types that are not
>
>> defined in chukwa-demux-config file.
>
>>
>
>> Can this be changed through some config file?
>
>>
>
>>
>
>>
>
>> I looked through the code and saw that TSProcessor is hard coded as
>> default
>
>> demuxer.
>
>>
>
>> I am using chukwa 0.4
>
>>
>
>>
>
>>
>
>> Regards
>
>>
>
>> Jaydeep
>
>>
>
>>
>
>>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.