-Hadoop idioms for reporting cluster and counter stats.
Jay Vyas 2012-08-16, 17:07
Hi guys : I want to start automating the output of counter stats, cluster
size, etc... at the end of the main map reduce jobs which we run. Is there
a simple way to do this ?
Here is my current thought :
1) Run all jobs from a driver class (we already do this).
2) At the end of each job, intercept the global counters and write them out
to a text file. This would
presumably be on the local fs.
3) Export the local filesystem.
4) Maybe the NameNode also has access to such data , maybe via an API
(clearly, the hadoop web ui gets this
data from somewhere, re in the "cluster summary" header..