Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # user >> Hadoop metric are not collected on amazon EMR


Copy link to this message
-
Re: Hadoop metric are not collected on amazon EMR
Hi Meena,

Hadoop 0.20.205 and Chukwa 0.4 is a mismatch.  Hadoop 0.20.20x has
been modified to use MetricsSink instead of MentricsContext.
Therefore, you will not see Hadoop metrics by using Chukwa 0.4.  You
will need Chukwa 0.5 to monitor Hadoop 0.20.205+.  In addition, you
should copy hadoop-metrics2.properties to $HADOOP_CONF_DIR to monitor
Hadoop.  Hope this helps.

regards,
Eric

On Tue, Apr 3, 2012 at 3:33 AM, Meena K.S <[EMAIL PROTECTED]> wrote:
> Hi,
>
>
>
> We want to use chukwa for hadoop log collection. We are running hive
> queries on EMR. So for the purpose of experimenting with chukwa we
> configured chukwa on a 2 node EMR cluster.  The collector and the
> agent are running and also the system logs are generated but the
> hadoop mapred and rpc /jvm logs are not being generated. Chukwa
> version : 0.4 and the  hadoop version :0.20.205
>
>
>
> My hadoop-metrics.properties is as below :
>
>
> # Licensed to the Apache Software Foundation (ASF) under one or more
>
> # contributor license agreements.  See the NOTICE file distributed with
>
> # this work for additional information regarding copyright ownership.
>
> # The ASF licenses this file to You under the Apache License, Version 2.0
>
> # (the "License"); you may not use this file except in compliance with
>
> # the License.  You may obtain a copy of the License at
>
> #
>
> #     http://www.apache.org/licenses/LICENSE-2.0
>
> #
>
> # Unless required by applicable law or agreed to in writing, software
>
> # distributed under the License is distributed on an "AS IS" BASIS,
>
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>
> # See the License for the specific language governing permissions and
>
> # limitations under the License.
>
>
>
> log4j.appender.chukwa.rpc.recordType=HadoopMetricsProcessor
>
> log4j.appender.chukwa.rpc.chukwaClientHostname=localhost
>
> log4j.appender.chukwa.rpc.chukwaClientPortNum=9093
>
> log4j.appender.chukwa.rpc.DatePattern=.yyyy-MM-dd
>
> log4j.appender.chukwa.rpc.layout=org.apache.log4j.PatternLayout
>
> log4j.appender.chukwa.rpc.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
>
> log4j.appender.chukwa.rpc.Dir=/tmp/chukwa/log/metrics
>
>
>
> log4j.appender.chukwa.dfs.recordType=HadoopMetricsProcessor
>
> log4j.appender.chukwa.dfs.chukwaClientHostname=localhost
>
> log4j.appender.chukwa.dfs.chukwaClientPortNum=9093
>
> log4j.appender.chukwa.dfs.DatePattern=.yyyy-MM-dd
>
> log4j.appender.chukwa.dfs.layout=org.apache.log4j.PatternLayout
>
> log4j.appender.chukwa.dfs.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
>
> log4j.appender.chukwa.dfs.Dir=/tmp/chukwa/log/metrics
>
>
>
> log4j.appender.chukwa.mapred.recordType=HadoopMetricsProcessor
>
> log4j.appender.chukwa.mapred.chukwaClientHostname=localhost
>
> log4j.appender.chukwa.mapred.chukwaClientPortNum=9093
>
> log4j.appender.chukwa.mapred.DatePattern=.yyyy-MM-dd
>
> log4j.appender.chukwa.mapred.layout=org.apache.log4j.PatternLayout
>
> log4j.appender.chukwa.mapred.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
>
> log4j.appender.chukwa.mapred.Dir=/tmp/chukwa/log/metrics
>
>
>
> I have also copied chukwa-hadoop-*-client.jar and json.jar to
> HADOOP_HOME/lib, and restarted hadoop, but still the logs were not
> generated.
>
> Also when I run telnet localhost 9093, the jvm or other hadoop metrics
> are not running, but the other system adapters are running correctly.
>
> Can someone help me here..
>
> Thanks
> Meena