HDFS doesnot support Appending i think . I m not sure about pig , if you are
using Hadoop directly you can zip the files and use zip as the input the
On Fri, Jun 17, 2011 at 6:56 AM, Xiaobo Gu <[EMAIL PROTECTED]> wrote:
> please refer to FileUtil.CopyMerge
> On Fri, Jun 17, 2011 at 8:33 AM, jagaran das <[EMAIL PROTECTED]>
> > Hi,
> > We have a requirement where
> > There would be huge number of small files to be pushed to hdfs and then
> use pig
> > to do analysis.
> > To get around the classic "Small File Issue" we merge the files and push
> > bigger file in to HDFS.
> > But we are loosing time in this merging process of our pipeline.
> > But If we can directly append to an existing file in HDFS we can save
> > "Merging Files" time.
> > Can you please suggest if there a newer stable version of Hadoop where
> can go
> > for appending ?
> > Thanks and Regards,
> > Jagaran