Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Hadoop idioms for reporting cluster and counter stats.


Copy link to this message
-
Hadoop idioms for reporting cluster and counter stats.
Hi guys : I want to start automating the output of  counter stats, cluster
size, etc... at the end of the main map reduce jobs which we run.  Is there
a simple way to do this ?

Here is my current thought :

1) Run all jobs from a driver class (we already do this).

2) At the end of each job, intercept the global counters and write them out
to a text file.  This would
presumably be on the local fs.

3) Export the local filesystem.

4) Maybe the NameNode also has access to such data , maybe via an API
(clearly, the hadoop web ui gets this
data from somewhere, re in the "cluster summary" header..
--
Jay Vyas
MMSB/UCHC
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB