Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Hadoop tool-kit for monitoring

Copy link to this message
Re: Hadoop tool-kit for monitoring

On May 17, 2011, at 3:11 PM, Mark question wrote:

> So what other memory consumption tools do you suggest? I don't want to do it
> manually and dump statistics into file because IO will affect performance
> too.

We watch memory with Ganglia.  We also tune our systems such that a task will only take X amount.  In other words, given an 8gb RAM:

1gb for the OS
1gb for the TT and DN
6gb for all tasks

if we assume each task will take max 1gb, then we end up with 3 maps and 3 reducers.  

Keep in mind that the mem consumed is more than just JVM heap size.