Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Reading while appending in 0.21


Copy link to this message
-
Re: Reading while appending in 0.21
This semantics is stated in our append specification attached to HDFS-265.
Hflushed data are visible to new readers who open the file after the hflush.

Hairong

On Mon, Jan 11, 2010 at 5:33 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> Thanks for the answer Konstantin, good to know. I also see that the
> commenter doesn't seem pleased with the fact that the file has to be
> reopened ;)
>
> So basically what I wrongly expected was a tailing feature. Searches
> in Jira don't give me a hit, would it make sense to open one?
>
> In the mean time, I'm ok with recreating the reader.
>
> Thx,
>
> J-D
>
> On Mon, Jan 11, 2010 at 5:15 PM, Konstantin Boudnik <[EMAIL PROTECTED]>
> wrote:
> > Jean,
> >
> > I believe this is how it has been intended to work. If you take a look at
> > the test src/test/hdfs/org/apache/hadoop/hdfs/TestHFlush.java in HDFS
> > workspace you'll a comment on this particular aspect in the line 124.
> >
> > Hope it helps,
> >  Cos
> >
> > On 1/11/10 16:42 , Jean-Daniel Cryans wrote:
> >>
> >> Hi,
> >>
> >> I'm trying to use the new hflush function from 0.21 so that a
> >> SequenceFile.Reader could read edits from a SequenceFile.Writer after
> >> a signal on a Condition. If I:
> >>
> >> create the Writer
> >> append entries
> >> hflush
> >> create the Reader
> >> next() through the entries
> >>
> >> It works fine. But after that if I only next(), using the same reader,
> >> after appending/hflush I won't see the new edits. But, if I create a
> >> new Reader after calling hflush, it works fine.
> >>
> >> So this does not work:
> >>
> >> create the Writer
> >> append entries
> >> hflush
> >> create the Reader
> >> next() through the entries
> >> append entries
> >> hflush
> >> next() through the entries
> >> append entries
> >> hflush
> >> next() through the entries
> >>
> >> This does:
> >>
> >> create the Writer
> >> append entries
> >> hflush
> >> create the Reader
> >> next() through the entries
> >> append entries
> >> hflush
> >> create a new Reader
> >> seek
> >> next() through the entries
> >>
> >> Is that the intended behavior?
> >>
> >> Thx!
> >>
> >> J-D
> >
>