*FYI this is a proof of concept cluster*

In my two node cluster that consists of
Master - Jobtracker, Datanode, Namenode, tasktracker, Secondarynamenode
And
Slave - Datenode , tasktraker

I have no more than 8g of ram on my slave and even less on the master and I am currently running 4 tasks on the slave and 2 on the master. My issue is that: is there a way where I can make sure that no more than 1g per thread  is allocated to read a large input file for my job?

Thanks,
SB

Per thread or per task?

On 4/13/12 2:17 PM, "Barry, Sean F" <[EMAIL PROTECTED]> wrote:

>*FYI this is a proof of concept cluster*
>
>In my two node cluster that consists of
>Master - Jobtracker, Datanode, Namenode, tasktracker, Secondarynamenode
>And
>Slave - Datenode , tasktraker
>
>I have no more than 8g of ram on my slave and even less on the master and
>I am currently running 4 tasks on the slave and 2 on the master. My issue
>is that: is there a way where I can make sure that no more than 1g per
>thread  is allocated to read a large input file for my job?
>
>Thanks,
>SB

Isnt essentially task = thread within my java app? Or maybe I am mistaken.

My main concern is that I need an upper bound to limit the memory footprint for each java thread. Are there any configurations in Hadoop that can help me do so?

-SB

-----Original Message-----
From: Serge Blazhievsky [mailto:[EMAIL PROTECTED]]
Sent: Friday, April 13, 2012 2:20 PM
To: [EMAIL PROTECTED]
Subject: Re: 1gb allocated per thread for input read

Per thread or per task?

On 4/13/12 2:17 PM, "Barry, Sean F" <[EMAIL PROTECTED]> wrote:

>*FYI this is a proof of concept cluster*
>
>In my two node cluster that consists of Master - Jobtracker, Datanode,
>Namenode, tasktracker, Secondarynamenode And Slave - Datenode ,
>tasktraker
>
>I have no more than 8g of ram on my slave and even less on the master
>and I am currently running 4 tasks on the slave and 2 on the master. My
>issue is that: is there a way where I can make sure that no more than
>1g per thread  is allocated to read a large input file for my job?
>
>Thanks,
>SB

Yes.

Take a look at
http://hadoop.apache.org/common/docs/current/mapred-default.html
You can pass parameters via mapred.child.java.opts to task process and
each task will obey
Regards,
Serge Blazhievsky

On 4/13/12 2:41 PM, "Barry, Sean F" <[EMAIL PROTECTED]> wrote:

>Isnt essentially task = thread within my java app? Or maybe I am mistaken.
>
>My main concern is that I need an upper bound to limit the memory
>footprint for each java thread. Are there any configurations in Hadoop
>that can help me do so?
>
>-SB
>
>-----Original Message-----
>From: Serge Blazhievsky [mailto:[EMAIL PROTECTED]]
>Sent: Friday, April 13, 2012 2:20 PM
>To: [EMAIL PROTECTED]
>Subject: Re: 1gb allocated per thread for input read
>
>Per thread or per task?
>
>
>
>On 4/13/12 2:17 PM, "Barry, Sean F" <[EMAIL PROTECTED]> wrote:
>
>>*FYI this is a proof of concept cluster*
>>
>>In my two node cluster that consists of Master - Jobtracker, Datanode,
>>Namenode, tasktracker, Secondarynamenode And Slave - Datenode ,
>>tasktraker
>>
>>I have no more than 8g of ram on my slave and even less on the master
>>and I am currently running 4 tasks on the slave and 2 on the master. My
>>issue is that: is there a way where I can make sure that no more than
>>1g per thread  is allocated to read a large input file for my job?
>>
>>Thanks,
>>SB
>