Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Running map tasks after all reduces have finished


+
Jan Lukavský 2012-08-23, 09:25
Copy link to this message
-
Re: Running map tasks after all reduces have finished
Hey Jan,

What version/distribution of Hadoop are you noticing this on?

On Thu, Aug 23, 2012 at 2:55 PM, Jan Lukavský
<[EMAIL PROTECTED]> wrote:
> Hi all,
>
> we are seeing strange behaviour of JobTracker in the following scenario:
>  - job finishes map phase and starts reduce
>  - after the shuffle phase of all reducers we loose a tasktracker, that
> doesn't run any reducer - so all remaining reducers are still running in the
> reduce phase
>  - map tasks that were running on the lost tasktracker are rescheduled
>  - reduces may finish earlier than the rescheduled map tasks and so the job
> is blocked waiting for the maps to finish, although their output is simple
> discarded
>
> Is this behaviour a bug or feature? :) I haven't found any JIRA that would
> describe it, if there exists one can anyone point me out?
>
> Thanks,
>  Jan
>

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB