Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Merge Reducers Outputs


Copy link to this message
-
Re: Merge Reducers Outputs
On 07/26/2011 06:52 PM, Mohamed Riadh Trad wrote:
> Dear All,
>
> Is it possible to set up a task with multiple reducers and merge reducers outputs into one single file?
>
> Bests,
>
> Trad Mohamed Riadh, M.Sc, Ing.

Not within the map-reduce job, but you can merge it after the job is
done.  At my previous company we used FileUtil.copyMerge() to do this,
and it worked quite well.

See:

http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FileUtil.html#copyMerge%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path,%20boolean,%20org.apache.hadoop.conf.Configuration,%20java.lang.String%29

DR
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB