Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Append MR output file to an exitsted HDFS file


+
YouPeng Yang 2013-04-22, 02:54
Copy link to this message
-
Re: Append MR output file to an exitsted HDFS file
I might be misunderstanding, but if you want each Reducer to append its
outputs to outputs to corresponding files that already exist in HDFS?

Remember that the reducers usually are outputting globs so you will have
several parts to your output - so the append has to be done in a way where
new reducer paritions corresponds to a old paritions.

If so - maybe you could play with your own OutputFormat by taking the
source from one that serves as a starting point, and replacing the
"create..." stream part with a call to write() with a call to append().

The reason this is tricky is that each OutputFormat is going to have to
find the corresponding file to append.
On Sun, Apr 21, 2013 at 10:54 PM, YouPeng Yang <[EMAIL PROTECTED]>wrote:

> Hi All
>
>      Can I append a MR output file to an existed file on HDFS.
>
>      I‘m using CDH4.1.2 vs MRv2
>
>
> Regards
>

--
Jay Vyas
http://jayunit100.blogspot.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB