Mark question 2011-05-17, 20:01
Allen Wittenauer 2011-05-17, 21:58
Mark question 2011-05-17, 22:11
On May 17, 2011, at 3:11 PM, Mark question wrote:
> So what other memory consumption tools do you suggest? I don't want to do it
> manually and dump statistics into file because IO will affect performance
We watch memory with Ganglia. We also tune our systems such that a task will only take X amount. In other words, given an 8gb RAM:
1gb for the OS
1gb for the TT and DN
6gb for all tasks
if we assume each task will take max 1gb, then we end up with 3 maps and 3 reducers.
Keep in mind that the mem consumed is more than just JVM heap size.
Konstantin Boudnik 2011-05-17, 22:16
Mark question 2011-05-17, 22:54