Hi,

1 - Hadoop MR contains a TaskMemoryManagerThread class that is used to
manage memory usage of tasks running under a TaskTracker. Why Hadoop
MR needs a class to manage memory? Why it couldn't rely on the JVM, or
this class is here for another purpose?

2 - How the JT knows that a Map or Reduce Task finished? Is through
the heartbeat?

Thanks

--
Pedro

For 1, TMMT uses ProcessTree to check for task that is running beyond
memory-limits and kills it.

On Thu, Dec 9, 2010 at 3:05 AM, Pedro Costa <[EMAIL PROTECTED]> wrote:

> Hi,
>
> 1 - Hadoop MR contains a TaskMemoryManagerThread class that is used to
> manage memory usage of tasks running under a TaskTracker. Why Hadoop
> MR needs a class to manage memory? Why it couldn't rely on the JVM, or
> this class is here for another purpose?
>
> 2 - How the JT knows that a Map or Reduce Task finished? Is through
> the heartbeat?
>
> Thanks
>
> --
> Pedro
>

> 2 - How the JT knows that a Map or Reduce Task finished? Is through
> the heartbeat?

Exactly.  Tasks communicate with their TTs through the umbilical, and each
TT communicates with the JT via heartbeat (and heartbeat response).

Greg

Hi,

On Thu, Dec 9, 2010 at 4:35 PM, Pedro Costa <[EMAIL PROTECTED]> wrote:
> Hi,
>
> 1 - Hadoop MR contains a TaskMemoryManagerThread class that is used to
> manage memory usage of tasks running under a TaskTracker. Why Hadoop
> MR needs a class to manage memory? Why it couldn't rely on the JVM, or
> this class is here for another purpose?
>

There are streaming and pipes map/reduce applications that launch
native processes from the map/reduce tasks that are outside the
control of the JVM. Indeed, even regular Java map/reduce programs
could fork/exec other programs. All of these processes could consume
memory that would not be accounted for if we relied only on the JVM to
get the memory usage. Hence a separate class that looks at the entire
process tree of the map/reduce task to account for memory consumed.

> 2 - How the JT knows that a Map or Reduce Task finished? Is through
> the heartbeat?
>

Yes.

> Thanks
>
> --
> Pedro
>