Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Is append allowed in HDFS?


Copy link to this message
-
Re: Is append allowed in HDFS?
Your approach looks fine to me. I'd throw in some
recovery/resume-from-errors-at-DN code around general tail-like
consumption but I think you may have already done that :)

But just for my curiosity - do you call sync for every record/unit or
batch it by a few, for your problem?

On Mon, Apr 9, 2012 at 10:34 PM, Inder Pall <[EMAIL PROTECTED]> wrote:
> Yes makes sense. My use-case is more like a producer/consumer and consumer
> trying to stream data as it arrives.
> Has anyone hit this before and if so resolved it in a better way.
>
> Apologies, if i am digressing from the subject of this thread however seems
> to land in the bucket of append support in HDFS.
>
> - Inder
>
>
> On Mon, Apr 9, 2012 at 6:27 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> Inder,
>>
>> Yes, that is a requirement for readers of sync-ing data. The new meta
>> entries can only be read by new readers. The read code would end up
>> being exactly like the implementation for method "fs -tail" at
>>
>> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/core/org/apache/hadoop/fs/FsShell.java?view=markup
>> (Line 1101)
>>
>> HBase does not read the WAL (HLog) continuously/vigorously as it
>> syncs, by the way. It only reads the them when a specific request is
>> made (for splitting, replaying and debug-printing).
>>
>> On Mon, Apr 9, 2012 at 6:05 PM, Inder Pall <[EMAIL PROTECTED]> wrote:
>> > Based on what i have tried, after a sync you need to open a new Reader.
>> > Please correct if that's not the write semantics.
>> >
>> > Thanks,
>> > - Inder
>> >
>> >
>> > On Mon, Apr 9, 2012 at 4:23 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>> >>
>> >> I'd also like to note that there are some unresolved issues with the
>> >> append version in the 1.x (stable) line.
>> >>
>> >> Note that HBase's use of the 0.20-append branch features are limited
>> >> to using "sync" calls alone (Described in p68 "Coherency Model",
>> >> Chapter 3 (The Hadoop Distributed File System) in Hadoop: The
>> >> Definitive Guide, 2nd Edition (O'Reilly)). Not the file-reopening
>> >> "append" calls. The latter is what is still with issues in the 1.x
>> >> releases today. Using the former is alright if its done in the way
>> >> similar to HBase's WAL (HLog) (or for similar needs).
>> >>
>> >> On Mon, Apr 9, 2012 at 3:45 PM, Ioan Eugen Stan <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > 2012/4/7 Florin P <[EMAIL PROTECTED]>:
>> >> >> Hello!
>> >> >>   Just google it for supporting of append into HDFS files and the
>> >> >> result:
>> >> >> I'm puzzled. Can someone say: YES you can append in TextFile or
>> >> >> SequenceFile
>> >> >> or whatever format. If yes, in which version this feature is
>> >> >> supported
>> >> >> ?
>> >> >> Also where can I find a good example of using the API? I know that
>> >> >> is a
>> >> >> long
>> >> >> debate about this subject, but really it is challenge to find on the
>> >> >> google
>> >> >> the current status of this feature.
>> >> >> I look forward for a trust source answer.
>> >> >> Thank you,
>> >> >>   Regards,
>> >> >>    Florin
>> >> >
>> >> > Hi Florian,
>> >> >
>> >> > HDFS supports append in Hadoop 1.0.x branch and also 0.22 (a.k.a
>> >> > hadoop 2.x branch) and 0.23 (a.k.a hadoop 3.x branch).
>> >> >
>> >> > [1] http://hbase.apache.org/book/hadoop.html
>> >> > [2] http://hbase.apache.org/book/hadoop.html -- search for append in
>> >> > release notes
>> >> >
>> >> > Cheers,
>> >> > --
>> >> > Ioan Eugen Stan
>> >> > http://ieugen.blogspot.com/
>> >>
>> >>
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>> >
>> >
>> > --
>> > Thanks,
>> > - Inder
>> >   Tech Platforms @Inmobi
>> >   Linkedin - http://goo.gl/eR4Ub
>>
>>
>>
>> --
>> Harsh J
>
>
>
>
> --
> Thanks,
> - Inder
>   Tech Platforms @Inmobi
>   Linkedin - http://goo.gl/eR4Ub

--
Harsh J