Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> metrics all use the same namespace


Copy link to this message
-
metrics all use the same namespace
Hi!

There is fixed issue in hadoop saying: "jvm metrics all use the same
namespace" - https://issues.apache.org/jira/browse/HADOOP-7507

I was able to apply this fix in our cluster using following line in
hadoop-metrics2.properties:
datanode.sink.ganglia.tagsForPrefix.jvm=*

So now I could see datanode jvm metrics with the following name, e.g.:
jvm.JvmMetrics.ProcessName_DataNode.GcCount

Also, it works for rpc namespace metrics, but looks little bit ugly because
of port number instead of service name:
rpc.rpc.port_50020.CallQueueLength

But it is not working for metricssystem and rpcdetails namespaces metrics
for example and may be other namespaces.
Configuration line we have:
datanode.sink.ganglia.tagsForPrefix.rpcdetailed=*
datanode.sink.ganglia.tagsForPrefix.metricssystem=*

The metrics name I could see , e.g.:
metricssystem.MetricsSystem.DroppedPubAll
rpcdetailed.rpcdetailed.InitReplicaRecoveryAvgTime

And from these names it is unclear which service sent it.
DataNode log shows following:

2013-01-17 15:01:36,706 DEBUG
org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31: Emitting metric
metricssystem.MetricsSystem.Sink_gangliaAvgTime, type double, value 77.0,
slope both from hostname node07.dom.local
2013-01-17 15:01:36,706 DEBUG
org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31: Emitting metric
metricssystem.MetricsSystem.NumAllSources, type int32, value 5, slope both
from hostname node07.dom.local
2013-01-17 15:01:36,706 DEBUG
org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31: Emitting metric
metricssystem.MetricsSystem.PublishNumOps, type float, value 57, slope
positive from hostname node07.dom.local
2013-01-17 15:01:36,706 DEBUG
org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31: Emitting metric
metricssystem.MetricsSystem.NumActiveSources, type int32, value 5, slope
both from hostname node07.dom.local

So, is it configuration issue, bug or something else?

Thanks in advance!

Hadoop version we use:
- MapReduce 2.0.0-mr1-cdh4.1.1
- HDFS 2.0.0-cdh4.1.1

--
Best Regards
Ivan Tretyakov
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB