Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Is append allowed in HDFS?


Copy link to this message
-
Re: Is append allowed in HDFS?
Inder,

Yes, that is a requirement for readers of sync-ing data. The new meta
entries can only be read by new readers. The read code would end up
being exactly like the implementation for method "fs -tail" at
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/core/org/apache/hadoop/fs/FsShell.java?view=markup
(Line 1101)

HBase does not read the WAL (HLog) continuously/vigorously as it
syncs, by the way. It only reads the them when a specific request is
made (for splitting, replaying and debug-printing).

On Mon, Apr 9, 2012 at 6:05 PM, Inder Pall <[EMAIL PROTECTED]> wrote:
> Based on what i have tried, after a sync you need to open a new Reader.
> Please correct if that's not the write semantics.
>
> Thanks,
> - Inder
>
>
> On Mon, Apr 9, 2012 at 4:23 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> I'd also like to note that there are some unresolved issues with the
>> append version in the 1.x (stable) line.
>>
>> Note that HBase's use of the 0.20-append branch features are limited
>> to using "sync" calls alone (Described in p68 "Coherency Model",
>> Chapter 3 (The Hadoop Distributed File System) in Hadoop: The
>> Definitive Guide, 2nd Edition (O'Reilly)). Not the file-reopening
>> "append" calls. The latter is what is still with issues in the 1.x
>> releases today. Using the former is alright if its done in the way
>> similar to HBase's WAL (HLog) (or for similar needs).
>>
>> On Mon, Apr 9, 2012 at 3:45 PM, Ioan Eugen Stan <[EMAIL PROTECTED]>
>> wrote:
>> > 2012/4/7 Florin P <[EMAIL PROTECTED]>:
>> >> Hello!
>> >>   Just google it for supporting of append into HDFS files and the
>> >> result:
>> >> I'm puzzled. Can someone say: YES you can append in TextFile or
>> >> SequenceFile
>> >> or whatever format. If yes, in which version this feature is supported
>> >> ?
>> >> Also where can I find a good example of using the API? I know that is a
>> >> long
>> >> debate about this subject, but really it is challenge to find on the
>> >> google
>> >> the current status of this feature.
>> >> I look forward for a trust source answer.
>> >> Thank you,
>> >>   Regards,
>> >>    Florin
>> >
>> > Hi Florian,
>> >
>> > HDFS supports append in Hadoop 1.0.x branch and also 0.22 (a.k.a
>> > hadoop 2.x branch) and 0.23 (a.k.a hadoop 3.x branch).
>> >
>> > [1] http://hbase.apache.org/book/hadoop.html
>> > [2] http://hbase.apache.org/book/hadoop.html -- search for append in
>> > release notes
>> >
>> > Cheers,
>> > --
>> > Ioan Eugen Stan
>> > http://ieugen.blogspot.com/
>>
>>
>>
>> --
>> Harsh J
>
>
>
>
> --
> Thanks,
> - Inder
>   Tech Platforms @Inmobi
>   Linkedin - http://goo.gl/eR4Ub

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB