Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - move tasks to another machine on the fly


Copy link to this message
-
Re: move tasks to another machine on the fly
Sharad Agarwal 2009-05-07, 11:43


> Just one more question, does Hadoop handles reassign of task failure
> to different machines in some way?
Yes. If task fails then it is retried, preferably on a different machine.
>
>
> I saw that sometimes, usually at the end, when there are more
> "processing units" available than map() tasks to process, the same
> map() tasks might be processed twice, then one is killed when the
> other finish first.
This is called speculative execution. Jobtracker monitors the progress
of tasks and if progress for an individual task is slow, it launches
another task. Whichever finishes first is used and other
one is killed.

- Sharad