Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Reading while appending in 0.21


Copy link to this message
-
Re: Reading while appending in 0.21
This semantics is stated in our append specification attached to HDFS-265.
Hflushed data are visible to new readers who open the file after the hflush.

Hairong

On Mon, Jan 11, 2010 at 5:33 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> Thanks for the answer Konstantin, good to know. I also see that the
> commenter doesn't seem pleased with the fact that the file has to be
> reopened ;)
>
> So basically what I wrongly expected was a tailing feature. Searches
> in Jira don't give me a hit, would it make sense to open one?
>
> In the mean time, I'm ok with recreating the reader.
>
> Thx,
>
> J-D
>
> On Mon, Jan 11, 2010 at 5:15 PM, Konstantin Boudnik <[EMAIL PROTECTED]>
> wrote:
> > Jean,
> >
> > I believe this is how it has been intended to work. If you take a look at
> > the test src/test/hdfs/org/apache/hadoop/hdfs/TestHFlush.java in HDFS
> > workspace you'll a comment on this particular aspect in the line 124.
> >
> > Hope it helps,
> >  Cos
> >
> > On 1/11/10 16:42 , Jean-Daniel Cryans wrote:
> >>
> >> Hi,
> >>
> >> I'm trying to use the new hflush function from 0.21 so that a
> >> SequenceFile.Reader could read edits from a SequenceFile.Writer after
> >> a signal on a Condition. If I:
> >>
> >> create the Writer
> >> append entries
> >> hflush
> >> create the Reader
> >> next() through the entries
> >>
> >> It works fine. But after that if I only next(), using the same reader,
> >> after appending/hflush I won't see the new edits. But, if I create a
> >> new Reader after calling hflush, it works fine.
> >>
> >> So this does not work:
> >>
> >> create the Writer
> >> append entries
> >> hflush
> >> create the Reader
> >> next() through the entries
> >> append entries
> >> hflush
> >> next() through the entries
> >> append entries
> >> hflush
> >> next() through the entries
> >>
> >> This does:
> >>
> >> create the Writer
> >> append entries
> >> hflush
> >> create the Reader
> >> next() through the entries
> >> append entries
> >> hflush
> >> create a new Reader
> >> seek
> >> next() through the entries
> >>
> >> Is that the intended behavior?
> >>
> >> Thx!
> >>
> >> J-D
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB