HDFS, mail # user - Append MR output file to an exitsted HDFS file
Append MR output file to an exitsted HDFS file
YouPeng Yang 2013-04-22, 02:54
Can I append a MR output file to an existed file on HDFS.
I‘m using CDH4.1.2 vs MRv2 Regards
Re: Append MR output file to an exitsted HDFS file
Jay Vyas 2013-04-22, 04:04
I might be misunderstanding, but if you want each Reducer to append its outputs to outputs to corresponding files that already exist in HDFS?
Remember that the reducers usually are outputting globs so you will have several parts to your output - so the append has to be done in a way where new reducer paritions corresponds to a old paritions.
If so - maybe you could play with your own OutputFormat by taking the source from one that serves as a starting point, and replacing the "create..." stream part with a call to write() with a call to append().
The reason this is tricky is that each OutputFormat is going to have to find the corresponding file to append. On Sun, Apr 21, 2013 at 10:54 PM, YouPeng Yang <[EMAIL PROTECTED]>wrote:
> Hi All > > Can I append a MR output file to an existed file on HDFS. > > I‘m using CDH4.1.2 vs MRv2 > > > Regards >