Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Re: hadoop file append


Copy link to this message
-
Re: hadoop file append
Harsh J 2013-03-18, 23:36
Appending on 1.x releases is available but not tested/supported and
can be toggled to be disabled completely. Appending works better on
2.x releases.

On Mon, Mar 18, 2013 at 9:14 PM, Tony Burton <[EMAIL PROTECTED]> wrote:
> Hi list,
>
>
>
> I’m using Hadoop 1.0.3 for a MapReduce task and I thought it might be a
> simple job to append a Counter value and some text to the end of a file
> (which ultimately will be in AWS S3). How wrong I was :)
>
>
>
> I’ve been reading about o.a.h.fs.FileSystem.append and whether it does or
> doesn’t work in my version of hadoop, and I'm not sure if it does or not.
> Can anyone say for definite either way? I'm going with the conclusion that
> it can't be trusted to perform as expected.
>
>
>
> What I'd like to do is this: Perform a daily MapReduce job and add up some
> some values along the way, and as well as the output of the MR job, add that
> day's date with the Counter value to the end of a file (in a different
> location), so the file grows by one line daily. Assuming
> o.a.h.fs.FileSystem.append is currently out of favour, do you have any
> suggestions for achieving this?
>
>
>
> Thanks!
>
>
>
> Tony
>
>
>
>
> *****************************************************************************
> P Please consider the environment before printing this email or attachments
>
>
> This email and any attachments are confidential, protected by copyright and
> may be legally privileged. If you are not the intended recipient, then the
> dissemination or copying of this email is prohibited. If you have received
> this in error, please notify the sender by replying by email and then delete
> the email completely from your system. Neither Sporting Index nor the sender
> accepts responsibility for any virus, or any other defect which might affect
> any computer or IT system into which the email is received and/or opened. It
> is the responsibility of the recipient to scan the email and no
> responsibility is accepted for any loss or damage arising in any way from
> receipt or use of this email. Sporting Index Ltd is a company registered in
> England and Wales with company number 2636842, whose registered office is at
> Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd is
> authorised and regulated by the UK Financial Services Authority (reg. no.
> 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any
> financial promotion contained herein has been issued and approved by
> Sporting Index Ltd.
>
>
> Outbound email has been scanned for viruses and SPAM

--
Harsh J