Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Hadoop tool-kit for monitoring

Copy link to this message
Re: Hadoop tool-kit for monitoring
Allen Wittenauer 2011-05-17, 22:15

On May 17, 2011, at 3:11 PM, Mark question wrote:

> So what other memory consumption tools do you suggest? I don't want to do it
> manually and dump statistics into file because IO will affect performance
> too.

We watch memory with Ganglia.  We also tune our systems such that a task will only take X amount.  In other words, given an 8gb RAM:

1gb for the OS
1gb for the TT and DN
6gb for all tasks

if we assume each task will take max 1gb, then we end up with 3 maps and 3 reducers.  

Keep in mind that the mem consumed is more than just JVM heap size.