Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa, mail # user - Hadoop metric are not collected on amazon EMR


+
meena k.s 2012-04-03, 11:10
+
Meena K.S 2012-04-03, 10:33
+
Eric Yang 2012-04-03, 16:06
+
meena k.s 2012-04-04, 08:03
+
Eric Yang 2012-04-05, 03:02
+
meena k.s 2012-04-04, 08:43
Copy link to this message
-
Re: Hadoop metric are not collected on amazon EMR
Ahmed Fathalla 2012-04-04, 11:09
It seems like you have problems accessing your HDFS. Are you sure you
configured it correctly? Can you access it via command line?

On Wed, Apr 4, 2012 at 10:43 AM, meena k.s <[EMAIL PROTECTED]>wrote:

> Hi Eric,
>
> Regarding my earlier query, i see the following error message in
> chukwa-collector.log
>
>  WARN main SeqFileWriter - Got an exception trying to rotate. Will try
> again in 300 seconds.
> org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> java.lang.NoSuchMethodException:
> org.apache.hadoop.hdfs.protocol.ClientProtocol.create(java.lang.String,
> org.apache.hadoop.fs.permission.FsPermission, java.lang.String, boolean,
> boolean, short, long)
>         at java.lang.Class.getMethod(Class.java:1605)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
> Thanks,
> Meena
>
> On Wed, Apr 4, 2012 at 1:33 PM, meena k.s <[EMAIL PROTECTED]>wrote:
>
>> Hi Eric,
>>
>> Thanks for the quick reply.I have tried to install chukwa 0.5
>> accordingly. but there seems to be a problem while starting
>> chukwa-collector. In the log files i could see something related to hbase.
>> I haven't installed any hbase in my cluster. Therefore I commented out the
>> parameters related to hbase in my chukwa-collector-conf.xml. Is it
>> necessary to have a hbase installation for chukwa 0.5 to work.
>>
>> Thanks,
>> Meena
>>
>> On Tue, Apr 3, 2012 at 9:36 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>
>>> Hi Meena,
>>>
>>> Hadoop 0.20.205 and Chukwa 0.4 is a mismatch.  Hadoop 0.20.20x has
>>> been modified to use MetricsSink instead of MentricsContext.
>>> Therefore, you will not see Hadoop metrics by using Chukwa 0.4.  You
>>> will need Chukwa 0.5 to monitor Hadoop 0.20.205+.  In addition, you
>>> should copy hadoop-metrics2.properties to $HADOOP_CONF_DIR to monitor
>>> Hadoop.  Hope this helps.
>>>
>>> regards,
>>> Eric
>>>
>>> On Tue, Apr 3, 2012 at 3:33 AM, Meena K.S <[EMAIL PROTECTED]> wrote:
>>> > Hi,
>>> >
>>> >
>>> >
>>> > We want to use chukwa for hadoop log collection. We are running hive
>>> > queries on EMR. So for the purpose of experimenting with chukwa we
>>> > configured chukwa on a 2 node EMR cluster.  The collector and the
>>> > agent are running and also the system logs are generated but the
>>> > hadoop mapred and rpc /jvm logs are not being generated. Chukwa
>>> > version : 0.4 and the  hadoop version :0.20.205
>>> >
>>> >
>>> >
>>> > My hadoop-metrics.properties is as below :
>>> >
>>> >
>>> > # Licensed to the Apache Software Foundation (ASF) under one or more
>>> >
>>> > # contributor license agreements.  See the NOTICE file distributed with
>>> >
>>> > # this work for additional information regarding copyright ownership.
>>> >
>>> > # The ASF licenses this file to You under the Apache License, Version
>>> 2.0
>>> >
>>> > # (the "License"); you may not use this file except in compliance with
>>> >
>>> > # the License.  You may obtain a copy of the License at
>>> >
>>> > #
>>> >
>>> > #     http://www.apache.org/licenses/LICENSE-2.0
>>> >
>>> > #
>>> >
>>> > # Unless required by applicable law or agreed to in writing, software
>>> >
>>> > # distributed under the License is distributed on an "AS IS" BASIS,
>>> >
>>> > # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>> implied.
>>> >
>>> > # See the License for the specific language governing permissions and
>>> >
>>> > # limitations under the License.
>>> >
>>> >
>>> >
>>> > log4j.appender.chukwa.rpc.recordType=HadoopMetricsProcessor
>>> >
>>> > log4j.appender.chukwa.rpc.chukwaClientHostname=localhost
>>> >
>>> > log4j.appender.chukwa.rpc.chukwaClientPortNum=9093
>>> >
>>> > log4j.appender.chukwa.rpc.DatePattern=.yyyy-MM-dd
>>> >
>>> > log4j.appender.chukwa.rpc.layout=org.apache.log4j.PatternLayout
Ahmed Fathalla
+
meena k.s 2012-04-04, 11:11
+
Eric Yang 2012-04-05, 03:04
+
meena k.s 2012-04-05, 05:12
+
Eric Yang 2012-04-05, 18:00