Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Hadoop in a Heterogeneous Environment - taking advantage of different processor specs


Copy link to this message
-
Hadoop in a Heterogeneous Environment - taking advantage of different processor specs
Hello,
Not sure if this has been asked or answered.
Suppose I have tasktrackers A1,A2,A3 each with 4 cores and 16GB ram.

mapred.tasktracker.map.tasks.maximum = 6
mapred.tasktracker.reduce.tasks.maximum = 4

Now suppose I have one more machine(X) with 8 cores and 32GB ram.

Since (if i'm not mistaken) tasktrackers talk to the jobtracker, can I take
advantage of X by
a) A1,A2,A3 each have the same hadoop-site.xml with the above values for
mapred.*
b.0) on B, i have a hadoop-site.xml with these valus

mapred.tasktracker.map.tasks.maximum = 5
mapred.tasktracker.reduce.tasks.maximum = 3

and start * one * tasktracker.
b.1) Then edit hadoop-site.xml, change the tasktracker port (if there is
such a thing) and

b.2) start *another* tasktracker.

We can skip b.1) if there is no such thing as a tasktracker port.
Hence I wil have /two/ tasktrackers running on X, one on Ai.  and thus take
advantage of X.

Is this at all possible? Or am i talking nonsense?

ANy pointers appreciated
Regards
Saptarshi
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB