Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Keeping Map-Tasks alive


Copy link to this message
-
Keeping Map-Tasks alive
Yaron Gonen 2012-08-05, 10:47
Hi,
Is there a way to keep a map-task alive after it has finished its work, to
later perform another task on its same input?
For example, consider the k-means clustering algorithm (k-means
description<http://en.wikipedia.org/wiki/K-means_clustering>and hadoop
implementation<http://codingwiththomas.blogspot.co.il/2011/05/k-means-clustering-with-mapreduce.html>).
The only thing changing between iterations is the clusters centers. All the
input points remain the same. Keeping the mapper alive, and performing the
next round of map-tasks on the same node will save a lot of communication
cost.

Thanks,
Yaron
+
Harsh J 2012-08-05, 16:49
+
Yaron Gonen 2012-08-05, 18:41
+
Harsh J 2012-08-05, 22:21
+
Yaron Gonen 2012-08-06, 07:23