Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Hadoop tool-kit for monitoring


+
Mark question 2011-05-17, 20:01
+
Allen Wittenauer 2011-05-17, 21:58
+
Mark question 2011-05-17, 22:11
Copy link to this message
-
Re: Hadoop tool-kit for monitoring

On May 17, 2011, at 3:11 PM, Mark question wrote:

> So what other memory consumption tools do you suggest? I don't want to do it
> manually and dump statistics into file because IO will affect performance
> too.

We watch memory with Ganglia.  We also tune our systems such that a task will only take X amount.  In other words, given an 8gb RAM:

1gb for the OS
1gb for the TT and DN
6gb for all tasks

if we assume each task will take max 1gb, then we end up with 3 maps and 3 reducers.  

Keep in mind that the mem consumed is more than just JVM heap size.
+
Konstantin Boudnik 2011-05-17, 22:16
+
Mark question 2011-05-17, 22:54
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB