Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Hadoop 2.2 Built-in Counters


Copy link to this message
-
Re: Hadoop 2.2 Built-in Counters
Hi,

This is the content of my shell script for running the job history server:

cd $HADOOP_PREFIX
hadoop fs -mkdir -p /mr-history/tmp
hadoop fs -chmod -R 1777 /mr-history/tmp
hadoop fs -mkdir -p /mr-history/done
hadoop fs -chmod -R 1777 /mr-history/done

sbin/mr-jobhistory-daemon.sh start historyserver

These configurable variables are in mapred-site.xml

<property>
        <name>mapreduce.jobhistory.address</name>
        <value>137.195.143.129:10020</value>
        <description>Default port is 10020.</description>
    </property>
    
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>137.195.143.129:19888</value>
        <description>Default port is 19888.</description>
    </property>

I start the history server on the same node as my resource manager
The counters are available from when the job is running from:

http://your-server:8088/proxy/application_1408007466921_0002/mapreduce/job/
job_1408007466921_0002

Drill down through the application master to the job.

If you donĀ¹t have the history server running the job data is not
persistent.

Hope this helps.
Regards,

Chris MacKenzie
telephone: 0131 332 6967
email: [EMAIL PROTECTED]
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
From:  ou senshaw <[EMAIL PROTECTED]>
Reply-To:  <[EMAIL PROTECTED]>
Date:  Thursday, 14 August 2014 07:14
To:  <[EMAIL PROTECTED]>
Subject:  Hadoop 2.2 Built-in Counters
Hi all,
I'm trying to analyze my mapreduce job performance via built-in counters
such as physical memory usage, heap memory usage...
When the job is running, I can watch these counters via Resource manager
website(namenode:8088). However, when the job is done, counter information
is not available in resource manager website anymore. I know I can get
them from client output. I was wondering if there is other place in name
node or data node to get the final counter measures regarding job id?
Thanks,
Shaw

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB