Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa >> mail # user >> Hadoop metric are not collected on amazon EMR


+
meena k.s 2012-04-03, 11:10
+
Meena K.S 2012-04-03, 10:33
+
Eric Yang 2012-04-03, 16:06
+
meena k.s 2012-04-04, 08:03
+
Eric Yang 2012-04-05, 03:02
+
meena k.s 2012-04-04, 08:43
+
Ahmed Fathalla 2012-04-04, 11:09
+
meena k.s 2012-04-04, 11:11
Copy link to this message
-
Re: Hadoop metric are not collected on amazon EMR
Which version of Hadoop are you using?  This looks like a mismatch
between how Chukwa
calls DFS client, and Hadoop jar file.

regards,
Eric

On Wed, Apr 4, 2012 at 1:43 AM, meena k.s <[EMAIL PROTECTED]> wrote:
> Hi Eric,
>
> Regarding my earlier query, i see the following error message in
> chukwa-collector.log
>
>  WARN main SeqFileWriter - Got an exception trying to rotate. Will try again
> in 300 seconds.
> org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> java.lang.NoSuchMethodException:
> org.apache.hadoop.hdfs.protocol.ClientProtocol.create(java.lang.String,
> org.apache.hadoop.fs.permission.FsPermission, java.lang.String, boolean,
> boolean, short, long)
>         at java.lang.Class.getMethod(Class.java:1605)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
> Thanks,
> Meena
>
> On Wed, Apr 4, 2012 at 1:33 PM, meena k.s <[EMAIL PROTECTED]> wrote:
>>
>> Hi Eric,
>>
>> Thanks for the quick reply.I have tried to install chukwa 0.5 accordingly.
>> but there seems to be a problem while starting chukwa-collector. In the log
>> files i could see something related to hbase. I haven't installed any hbase
>> in my cluster. Therefore I commented out the parameters related to hbase in
>> my chukwa-collector-conf.xml. Is it necessary to have a hbase installation
>> for chukwa 0.5 to work.
>>
>> Thanks,
>> Meena
>>
>> On Tue, Apr 3, 2012 at 9:36 PM, Eric Yang <[EMAIL PROTECTED]> wrote:
>>>
>>> Hi Meena,
>>>
>>> Hadoop 0.20.205 and Chukwa 0.4 is a mismatch.  Hadoop 0.20.20x has
>>> been modified to use MetricsSink instead of MentricsContext.
>>> Therefore, you will not see Hadoop metrics by using Chukwa 0.4.  You
>>> will need Chukwa 0.5 to monitor Hadoop 0.20.205+.  In addition, you
>>> should copy hadoop-metrics2.properties to $HADOOP_CONF_DIR to monitor
>>> Hadoop.  Hope this helps.
>>>
>>> regards,
>>> Eric
>>>
>>> On Tue, Apr 3, 2012 at 3:33 AM, Meena K.S <[EMAIL PROTECTED]> wrote:
>>> > Hi,
>>> >
>>> >
>>> >
>>> > We want to use chukwa for hadoop log collection. We are running hive
>>> > queries on EMR. So for the purpose of experimenting with chukwa we
>>> > configured chukwa on a 2 node EMR cluster.  The collector and the
>>> > agent are running and also the system logs are generated but the
>>> > hadoop mapred and rpc /jvm logs are not being generated. Chukwa
>>> > version : 0.4 and the  hadoop version :0.20.205
>>> >
>>> >
>>> >
>>> > My hadoop-metrics.properties is as below :
>>> >
>>> >
>>> > # Licensed to the Apache Software Foundation (ASF) under one or more
>>> >
>>> > # contributor license agreements.  See the NOTICE file distributed with
>>> >
>>> > # this work for additional information regarding copyright ownership.
>>> >
>>> > # The ASF licenses this file to You under the Apache License, Version
>>> > 2.0
>>> >
>>> > # (the "License"); you may not use this file except in compliance with
>>> >
>>> > # the License.  You may obtain a copy of the License at
>>> >
>>> > #
>>> >
>>> > #     http://www.apache.org/licenses/LICENSE-2.0
>>> >
>>> > #
>>> >
>>> > # Unless required by applicable law or agreed to in writing, software
>>> >
>>> > # distributed under the License is distributed on an "AS IS" BASIS,
>>> >
>>> > # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>> > implied.
>>> >
>>> > # See the License for the specific language governing permissions and
>>> >
>>> > # limitations under the License.
>>> >
>>> >
>>> >
>>> > log4j.appender.chukwa.rpc.recordType=HadoopMetricsProcessor
>>> >
>>> > log4j.appender.chukwa.rpc.chukwaClientHostname=localhost
>>> >
>>> > log4j.appender.chukwa.rpc.chukwaClientPortNum=9093
>>> >
>>> > log4j.appender.chukwa.rpc.DatePattern=.yyyy-MM-dd
>
+
meena k.s 2012-04-05, 05:12
+
Eric Yang 2012-04-05, 18:00
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB