Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # user >> Changing default demuxer


Copy link to this message
-
Re: Changing default demuxer
This is a bug in the documentation that I've noticed before. I've
opened a JIRA for it:
https://issues.apache.org/jira/browse/CHUKWA-528

The default processor is actually DefaultProcessor.

On Wed, Sep 29, 2010 at 5:43 AM, Stuti Awasthi
<[EMAIL PROTECTED]> wrote:
> Hi all,
>
>
>
> I have applied the patch and set the property in chukwa-demux-conf.xml file
> as follows :
>
> <property>
>
>  <name>chukwa.demux.mapper.default.processor</name>
>
>  <value>com.test.genericdemuxer.xyz</value>
>
> </property>
>
>
>
> <property>
>
>  <name>chukwa.demux.reducer.default.processor</name>
>
>  <value>com.test.genericdemuxer.xyz</value>
>
> </property>
>
> I tested the whole process from log collection to creation of .evt files in
> repos, that worked fine.
>
> Please note I have not added any datatype for my logtype in demux-conf file
> so that it can go to my processor that I have set in the property file.
>
>
>
> I also tested the whole process again without these entries in the
> demux-conf file to check the default behavior of chukwa, but surprisingly
> that also worked fine without any error.
>
>
>
> There was a difference in the sizes of the .evt file that generated from
> both the process. I am not able to figure out that whether my processor is
> getting used or not.
>
>
>
> How can I get to know that it is using my processor?
>
>
>
> I also have a query that in demux.java , we have code where
>  DefaultProcessor is hardcoded, and in documentation it is written that
> TSProcessor will the default processor will be used if we do not specify the
> datatype for particular log type in demux-con.xml file.
>
> String defaultProcessor = Demux.jobConf.get(
>
>             "chukwa.demux.mapper.default.processor",
>
>
> "org.apache.hadoop.chukwa.extraction.demux.processor.mapper.DefaultProcessor");
>
>
>
> String processorClass = Demux.jobConf.get(chunk.getDataType(),
>
>                 defaultProcessor);
>
>
>
> I think that either I am missing something or not able to clearly understand
> the flow.
>
> Please suggest.
>
>
>
> Regards
>
> Stuti Awasthi
>
>
>
> -----Original Message-----
>
> From: Bill Graham [mailto:[EMAIL PROTECTED]]
>
> Sent: Tuesday, September 28, 2010 9:32 AM
>
> To: [EMAIL PROTECTED]
>
> Cc: Jaydeep Ayachit
>
> Subject: Re: Changing default demuxer
>
>
>
> This can be changed with the following configs:
>
>
>
> chukwa.demux.mapper.default.processor
>
> chukwa.demux.reducer.default.processor
>
>
>
> Reference JIRA:
>
> https://issues.apache.org/jira/browse/CHUKWA-473
>
>
>
> This is part from Chukwa 0.4.0.
>
>
>
>
>
> On Mon, Sep 27, 2010 at 5:47 AM, Jaydeep Ayachit
>
> <[EMAIL PROTECTED]> wrote:
>
>> Hello,
>
>>
>
>>
>
>>
>
>> Chukwa by default uses TSProcessor demuxer for data types that are not
>
>> defined in chukwa-demux-config file.
>
>>
>
>> Can this be changed through some config file?
>
>>
>
>>
>
>>
>
>> I looked through the code and saw that TSProcessor is hard coded as
>> default
>
>> demuxer.
>
>>
>
>> I am using chukwa 0.4
>
>>
>
>>
>
>>
>
>> Regards
>
>>
>
>> Jaydeep
>
>>
>
>>
>
>>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB