Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - Metrics ..


+
Mark Olimpiati 2012-08-29, 19:54
+
Wong, David 2012-08-29, 20:28
Copy link to this message
-
Re: Metrics ..
Mark Olimpiati 2012-08-29, 20:47
Hi David,

   I  enabled the "jvm.class" of the hadoop-metrics.properties, you're
output seems to be from something else (dfs.class or mapred.class) which
reports hadoop deamons performace. For example your output shows
"processName=TaskTracker"
which I'm not looking for.

  How can I report jvm statistics for individual jvms (maps/reducers) ??

Thank you,
Mark

On Wed, Aug 29, 2012 at 1:28 PM, Wong, David (DMITS) <[EMAIL PROTECTED]> wrote:

> Here's a snippet of tasktracker metrics using Metrics2.  (I think there
> were (more) gaps in the pre-metrics2 versions.)
> Note that you'll need to have hadoop-env.sh and hadoop-metrics2.properties
> setup on all the nodes you want reports from.
>
> 1345570905436 ugi.ugi: context=ugi, hostName=sqws31.caclab.cac.cpqcorp.net,
> loginSuccess_num_ops=0, loginSuccess_avg_time=0.0, loginFailure_num_ops=0,
> loginFailure_avg_time=0.0
> 1345570905436 jvm.metrics: context=jvm, processName=TaskTracker,
> sessionId=, hostName=sqws31.caclab.cac.cpqcorp.net,
> memNonHeapUsedM=11.540627, memNonHeapCommittedM=18.25,
> memHeapUsedM=12.972412, memHeapCommittedM=61.375, gcCount=1,
> gcTimeMillis=6, threadsNew=0, threadsRunnable=9, threadsBlocked=0,
> threadsWaiting=9, threadsTimedWaiting=1, threadsTerminated=0, logFatal=0,
> logError=0, logWarn=0, logInfo=1
> 1345570905436 mapred.tasktracker: context=mapred, sessionId=, hostName> sqws31.caclab.cac.cpqcorp.net, maps_running=0, reduces_running=0,
> mapTaskSlots=2, reduceTaskSlots=2, tasks_completed=0,
> tasks_failed_timeout=0, tasks_failed_ping=0
> 1345570905436 rpcdetailed.rpcdetailed: context=rpcdetailed, port=33997,
> hostName=sqws31.caclab.cac.cpqcorp.net
> 1345570905436 rpc.rpc: context=rpc, port=33997, hostName> sqws31.caclab.cac.cpqcorp.net, rpcAuthenticationSuccesses=0,
> rpcAuthenticationFailures=0, rpcAuthorizationSuccesses=0,
> rpcAuthorizationFailures=0, ReceivedBytes=0, SentBytes=0,
> RpcQueueTime_num_ops=0, RpcQueueTime_avg_time=0.0,
> RpcProcessingTime_num_ops=0, RpcProcessingTime_avg_time=0.0,
> NumOpenConnections=0, callQueueLen=0
> 1345570905436 metricssystem.MetricsSystem: context=metricssystem, hostName> sqws31.caclab.cac.cpqcorp.net, num_sources=5, num_sinks=1,
> sink.file.latency_num_ops=0, sink.file.latency_avg_time=0.0,
> sink.file.dropped=0, sink.file.qsize=0, snapshot_num_ops=5,
> snapshot_avg_time=0.2, snapshot_stdev_time=0.447213595499958,
> snapshot_imin_time=0.0, snapshot_imax_time=1.0, snapshot_min_time=0.0,
> snapshot_max_time=1.0, publish_num_ops=0, publish_avg_time=0.0,
> publish_stdev_time=0.0, publish_imin_time=3.4028234663852886E38,
> publish_imax_time=1.401298464324817E-45,
> publish_min_time=3.4028234663852886E38,
> publish_max_time=1.401298464324817E-45, dropped_pub_all=0
> 1345570915435 ugi.ugi: context=ugi, hostName=sqws31.caclab.cac.cpqcorp.net
> 1345570915435 jvm.metrics: context=jvm, processName=TaskTracker,
> sessionId=, hostName=sqws31.caclab.cac.cpqcorp.net,
> memNonHeapUsedM=11.549316, memNonHeapCommittedM=18.25,
> memHeapUsedM=13.136337, memHeapCommittedM=61.375, gcCount=1,
> gcTimeMillis=6, threadsNew=0, threadsRunnable=9, threadsBlocked=0,
> threadsWaiting=9, threadsTimedWaiting=1, threadsTerminated=0, logFatal=0,
> logError=0, logWarn=0, logInfo=1
> 1345570915435 mapred.tasktracker: context=mapred, sessionId=, hostName> sqws31.caclab.cac.cpqcorp.net, maps_running=0, reduces_running=0,
> mapTaskSlots=2, reduceTaskSlots=2
> 1345570915435 rpcdetailed.rpcdetailed: context=rpcdetailed, port=33997,
> hostName=sqws31.caclab.cac.cpqcorp.net
> 1345570915435 rpc.rpc: context=rpc, port=33997, hostName> sqws31.caclab.cac.cpqcorp.net
> 1345570915435 metricssystem.MetricsSystem: context=metricssystem, hostName> sqws31.caclab.cac.cpqcorp.net, num_sources=5, num_sinks=1,
> sink.file.latency_num_ops=1, sink.file.latency_avg_time=4.0,
> snapshot_num_ops=11, snapshot_avg_time=0.16666666666666669,
> snapshot_stdev_time=0.408248290463863, snapshot_imin_time=0.0,
> snapshot_imax_time=1.0, snapshot_min_time=0.0, snapshot_max_time=1.0,