Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Re: knowing the nodes on which reduce tasks will run

Steve Loughran 2012-09-04, 11:33
Copy link to this message
Re: knowing the nodes on which reduce tasks will run
The short answer is no.
The longer answer is that you can attempt to force data locality, however even then if an open slot becomes available, its used regardless of what you want to do...
On Sep 3, 2012, at 9:19 AM, Abhay Ratnaparkhi <[EMAIL PROTECTED]> wrote:

> Hello,
> How can one get to know the nodes on which reduce tasks will run?
> One of my job is running and it's completing all the map tasks.
> My map tasks write lots of intermediate data. The intermediate directory is getting full on all the nodes.
> If the reduce task take any node from cluster then It'll try to copy the data to same disk and it'll eventually fail due to Disk space related exceptions.
> I have added few more tasktracker nodes in the cluster and now want to run reducer on new nodes only.
> Is it possible to choose a node on which the reducer will run? What's the algorithm hadoop uses to get a new node to run reducer?
> Thanks in advance.
> Bye
> Abhay
Hemanth Yamijala 2012-09-03, 15:56
Abhay Ratnaparkhi 2012-09-03, 16:00