Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Memory Manager in Hadoop MR


+
Pedro Costa 2010-12-09, 11:05
+
Ted Yu 2010-12-09, 18:17
+
Greg Roelofs 2010-12-10, 01:19
Copy link to this message
-
Re: Memory Manager in Hadoop MR
Hi,

On Thu, Dec 9, 2010 at 4:35 PM, Pedro Costa <[EMAIL PROTECTED]> wrote:
> Hi,
>
> 1 - Hadoop MR contains a TaskMemoryManagerThread class that is used to
> manage memory usage of tasks running under a TaskTracker. Why Hadoop
> MR needs a class to manage memory? Why it couldn't rely on the JVM, or
> this class is here for another purpose?
>

There are streaming and pipes map/reduce applications that launch
native processes from the map/reduce tasks that are outside the
control of the JVM. Indeed, even regular Java map/reduce programs
could fork/exec other programs. All of these processes could consume
memory that would not be accounted for if we relied only on the JVM to
get the memory usage. Hence a separate class that looks at the entire
process tree of the map/reduce task to account for memory consumed.

> 2 - How the JT knows that a Map or Reduce Task finished? Is through
> the heartbeat?
>

Yes.

> Thanks
>
> --
> Pedro
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB