Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Is there any way to set Reducer to output to multi-places?


Copy link to this message
-
答复: Is there any way to set Reducer to output to multi-places?
Thanks, Binglin

 

I found the class below that can do it :).

http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/lib/MultipleOutputs.html

 

 

 

发件人: Binglin Chang [mailto:[EMAIL PROTECTED]]
发送时间: Monday, September 02, 2013 17:37
收件人: [EMAIL PROTECTED]
主题: Re: Is there any way to set Reducer to output to multi-places?

 

MultipleOutputFormat allows you to write multiple files in one reducer, but can't write output to HDFS and Database concurrently, but I is a good example to show how you can write a customized OutputFormat to achieve this.

Please note that for fault tolerance, a reducer may run multiple times, this may generate redundant data, hadoop handles files using FileOutputCommitter, you need to handle database case by yourself(e.g. insert record only if record doesn't exists).

 

On Mon, Sep 2, 2013 at 5:11 PM, Rahul Bhattacharjee <[EMAIL PROTECTED]> wrote:

This might help

http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/lib/MultipleOutputFormat.html

Thanks,
Rahul

 

On Mon, Sep 2, 2013 at 2:38 PM, Francis.Hu <[EMAIL PROTECTED]> wrote:

hi, All

 

Is there any way to set Reducer to output to multi-places ?  For example: a reducer's result can be output to HDFS and Database concurrently.

 

Thanks,

Francis.Hu

 

 

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB