-Re: Hadoop in a Heterogeneous Environment - taking advantage of different processor specs
Saptarshi Guha 2009-07-28, 15:46
Tsk tsk, silly of me. Of course I could do that.
Thanks for the confirmation
On Tue, Jul 28, 2009 at 11:25 AM, Harish Mallipeddi <
[EMAIL PROTECTED]> wrote:
> Why do you want to start 2 tasktrackers on X? Just increase the no. of map
> & reduce slots on X and that should be enough to take advantage of the extra
> processing power on X.
> - Harish
> On Tue, Jul 28, 2009 at 8:10 PM, Saptarshi Guha <[EMAIL PROTECTED]>wrote:
>> Not sure if this has been asked or answered.
>> Suppose I have tasktrackers A1,A2,A3 each with 4 cores and 16GB ram.
>> mapred.tasktracker.map.tasks.maximum = 6
>> mapred.tasktracker.reduce.tasks.maximum = 4
>> Now suppose I have one more machine(X) with 8 cores and 32GB ram.
>> Since (if i'm not mistaken) tasktrackers talk to the jobtracker, can I
>> advantage of X by
>> a) A1,A2,A3 each have the same hadoop-site.xml with the above values for
>> b.0) on B, i have a hadoop-site.xml with these valus
>> mapred.tasktracker.map.tasks.maximum = 5
>> mapred.tasktracker.reduce.tasks.maximum = 3
>> and start * one * tasktracker.
>> b.1) Then edit hadoop-site.xml, change the tasktracker port (if there is
>> such a thing) and
>> b.2) start *another* tasktracker.
>> We can skip b.1) if there is no such thing as a tasktracker port.
>> Hence I wil have /two/ tasktrackers running on X, one on Ai. and thus
>> advantage of X.
>> Is this at all possible? Or am i talking nonsense?
>> ANy pointers appreciated
> Harish Mallipeddi