Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Hadoop in a Heterogeneous Environment - taking advantage of different processor specs


Copy link to this message
-
Re: Hadoop in a Heterogeneous Environment - taking advantage of different processor specs
Tsk tsk, silly of me. Of course I could do that.

Thanks for the confirmation
Regards
Saptarshi
On Tue, Jul 28, 2009 at 11:25 AM, Harish Mallipeddi <
[EMAIL PROTECTED]> wrote:

> Why do you want to start 2 tasktrackers on X? Just increase the no. of map
> & reduce slots on X and that should be enough to take advantage of the extra
> processing power on X.
>
> - Harish
>
>
> On Tue, Jul 28, 2009 at 8:10 PM, Saptarshi Guha <[EMAIL PROTECTED]>wrote:
>
>> Hello,
>> Not sure if this has been asked or answered.
>> Suppose I have tasktrackers A1,A2,A3 each with 4 cores and 16GB ram.
>>
>> mapred.tasktracker.map.tasks.maximum = 6
>> mapred.tasktracker.reduce.tasks.maximum = 4
>>
>> Now suppose I have one more machine(X) with 8 cores and 32GB ram.
>>
>> Since (if i'm not mistaken) tasktrackers talk to the jobtracker, can I
>> take
>> advantage of X by
>> a) A1,A2,A3 each have the same hadoop-site.xml with the above values for
>> mapred.*
>> b.0) on B, i have a hadoop-site.xml with these valus
>>
>> mapred.tasktracker.map.tasks.maximum = 5
>> mapred.tasktracker.reduce.tasks.maximum = 3
>>
>> and start * one * tasktracker.
>> b.1) Then edit hadoop-site.xml, change the tasktracker port (if there is
>> such a thing) and
>>
>> b.2) start *another* tasktracker.
>>
>> We can skip b.1) if there is no such thing as a tasktracker port.
>> Hence I wil have /two/ tasktrackers running on X, one on Ai.  and thus
>> take
>> advantage of X.
>>
>> Is this at all possible? Or am i talking nonsense?
>>
>> ANy pointers appreciated
>> Regards
>> Saptarshi
>>
>
>
>
> --
> Harish Mallipeddi
> http://blog.poundbang.in
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB