Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Submitting a Hadoop task from withing a reducer


Copy link to this message
-
Submitting a Hadoop task from withing a reducer
Yaron Gonen 2011-10-04, 15:45
Hi,
Hadoop tasks are always stacked to form a linear user-managed workflow (a
reduce step cannot start before all previous mappers have stopped etc). This
may be problematic in recursive tasks: for example in a BFS we will not get
any output until the longest branch has been reached.
In order to solve than, an idea came up of submitting a whole Hadoop task
from within a reducer. Have anyone tried it?
Thanks.