Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Hadoop in a Heterogeneous Environment - taking advantage of different processor specs


Copy link to this message
-
Re: Hadoop in a Heterogeneous Environment - taking advantage of different processor specs
Tsk tsk, silly of me. Of course I could do that.

Thanks for the confirmation
Regards
Saptarshi
On Tue, Jul 28, 2009 at 11:25 AM, Harish Mallipeddi <
[EMAIL PROTECTED]> wrote:

> Why do you want to start 2 tasktrackers on X? Just increase the no. of map
> & reduce slots on X and that should be enough to take advantage of the extra
> processing power on X.
>
> - Harish
>
>
> On Tue, Jul 28, 2009 at 8:10 PM, Saptarshi Guha <[EMAIL PROTECTED]>wrote:
>
>> Hello,
>> Not sure if this has been asked or answered.
>> Suppose I have tasktrackers A1,A2,A3 each with 4 cores and 16GB ram.
>>
>> mapred.tasktracker.map.tasks.maximum = 6
>> mapred.tasktracker.reduce.tasks.maximum = 4
>>
>> Now suppose I have one more machine(X) with 8 cores and 32GB ram.
>>
>> Since (if i'm not mistaken) tasktrackers talk to the jobtracker, can I
>> take
>> advantage of X by
>> a) A1,A2,A3 each have the same hadoop-site.xml with the above values for
>> mapred.*
>> b.0) on B, i have a hadoop-site.xml with these valus
>>
>> mapred.tasktracker.map.tasks.maximum = 5
>> mapred.tasktracker.reduce.tasks.maximum = 3
>>
>> and start * one * tasktracker.
>> b.1) Then edit hadoop-site.xml, change the tasktracker port (if there is
>> such a thing) and
>>
>> b.2) start *another* tasktracker.
>>
>> We can skip b.1) if there is no such thing as a tasktracker port.
>> Hence I wil have /two/ tasktrackers running on X, one on Ai.  and thus
>> take
>> advantage of X.
>>
>> Is this at all possible? Or am i talking nonsense?
>>
>> ANy pointers appreciated
>> Regards
>> Saptarshi
>>
>
>
>
> --
> Harish Mallipeddi
> http://blog.poundbang.in
>