Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa, mail # user - Hadoop metric are not collected on amazon EMR


Copy link to this message
-
Re: Hadoop metric are not collected on amazon EMR
Eric Yang 2012-04-05, 03:02
Hi Meena,

Chukwa 0.5.0 has HBase enabled by default.  You can modify it to
stream to HDFS by following the instructions on this page:

http://incubator.apache.org/chukwa/docs/r0.5.0/collector.html

regards,
Eric

On Wed, Apr 4, 2012 at 1:03 AM, meena k.s <[EMAIL PROTECTED]> wrote:
> Hi Eric,
>
> Thanks for the quick reply.I have tried to install chukwa 0.5 accordingly.
> but there seems to be a problem while starting chukwa-collector. In the log
> files i could see something related to hbase. I haven't installed any hbase
> in my cluster. Therefore I commented out the parameters related to hbase in
> my chukwa-collector-conf.xml. Is it necessary to have a hbase installation
> for chukwa 0.5 to work.
>
> Thanks,
> Meena
>
> On Tue, Apr 3, 2012 at 9:36 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>
>> Hi Meena,
>>
>> Hadoop 0.20.205 and Chukwa 0.4 is a mismatch.  Hadoop 0.20.20x has
>> been modified to use MetricsSink instead of MentricsContext.
>> Therefore, you will not see Hadoop metrics by using Chukwa 0.4.  You
>> will need Chukwa 0.5 to monitor Hadoop 0.20.205+.  In addition, you
>> should copy hadoop-metrics2.properties to $HADOOP_CONF_DIR to monitor
>> Hadoop.  Hope this helps.
>>
>> regards,
>> Eric
>>
>> On Tue, Apr 3, 2012 at 3:33 AM, Meena K.S <[EMAIL PROTECTED]> wrote:
>> > Hi,
>> >
>> >
>> >
>> > We want to use chukwa for hadoop log collection. We are running hive
>> > queries on EMR. So for the purpose of experimenting with chukwa we
>> > configured chukwa on a 2 node EMR cluster.  The collector and the
>> > agent are running and also the system logs are generated but the
>> > hadoop mapred and rpc /jvm logs are not being generated. Chukwa
>> > version : 0.4 and the  hadoop version :0.20.205
>> >
>> >
>> >
>> > My hadoop-metrics.properties is as below :
>> >
>> >
>> > # Licensed to the Apache Software Foundation (ASF) under one or more
>> >
>> > # contributor license agreements.  See the NOTICE file distributed with
>> >
>> > # this work for additional information regarding copyright ownership.
>> >
>> > # The ASF licenses this file to You under the Apache License, Version
>> > 2.0
>> >
>> > # (the "License"); you may not use this file except in compliance with
>> >
>> > # the License.  You may obtain a copy of the License at
>> >
>> > #
>> >
>> > #     http://www.apache.org/licenses/LICENSE-2.0
>> >
>> > #
>> >
>> > # Unless required by applicable law or agreed to in writing, software
>> >
>> > # distributed under the License is distributed on an "AS IS" BASIS,
>> >
>> > # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>> > implied.
>> >
>> > # See the License for the specific language governing permissions and
>> >
>> > # limitations under the License.
>> >
>> >
>> >
>> > log4j.appender.chukwa.rpc.recordType=HadoopMetricsProcessor
>> >
>> > log4j.appender.chukwa.rpc.chukwaClientHostname=localhost
>> >
>> > log4j.appender.chukwa.rpc.chukwaClientPortNum=9093
>> >
>> > log4j.appender.chukwa.rpc.DatePattern=.yyyy-MM-dd
>> >
>> > log4j.appender.chukwa.rpc.layout=org.apache.log4j.PatternLayout
>> >
>> > log4j.appender.chukwa.rpc.layout.ConversionPattern=%d{ISO8601} %p %c:
>> > %m%n
>> >
>> > log4j.appender.chukwa.rpc.Dir=/tmp/chukwa/log/metrics
>> >
>> >
>> >
>> > log4j.appender.chukwa.dfs.recordType=HadoopMetricsProcessor
>> >
>> > log4j.appender.chukwa.dfs.chukwaClientHostname=localhost
>> >
>> > log4j.appender.chukwa.dfs.chukwaClientPortNum=9093
>> >
>> > log4j.appender.chukwa.dfs.DatePattern=.yyyy-MM-dd
>> >
>> > log4j.appender.chukwa.dfs.layout=org.apache.log4j.PatternLayout
>> >
>> > log4j.appender.chukwa.dfs.layout.ConversionPattern=%d{ISO8601} %p %c:
>> > %m%n
>> >
>> > log4j.appender.chukwa.dfs.Dir=/tmp/chukwa/log/metrics
>> >
>> >
>> >
>> > log4j.appender.chukwa.mapred.recordType=HadoopMetricsProcessor
>> >
>> > log4j.appender.chukwa.mapred.chukwaClientHostname=localhost
>> >
>> > log4j.appender.chukwa.mapred.chukwaClientPortNum=9093
>> >
>> > log4j.appender.chukwa.mapred.DatePattern=.yyyy-MM-dd