Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Append MR output file to an exitsted HDFS file

Copy link to this message
Re: Append MR output file to an exitsted HDFS file
I might be misunderstanding, but if you want each Reducer to append its
outputs to outputs to corresponding files that already exist in HDFS?

Remember that the reducers usually are outputting globs so you will have
several parts to your output - so the append has to be done in a way where
new reducer paritions corresponds to a old paritions.

If so - maybe you could play with your own OutputFormat by taking the
source from one that serves as a starting point, and replacing the
"create..." stream part with a call to write() with a call to append().

The reason this is tricky is that each OutputFormat is going to have to
find the corresponding file to append.
On Sun, Apr 21, 2013 at 10:54 PM, YouPeng Yang <[EMAIL PROTECTED]>wrote:

> Hi All
>      Can I append a MR output file to an existed file on HDFS.
>      I‘m using CDH4.1.2 vs MRv2
> Regards

Jay Vyas