-Re: hflush not working for me?
stack 2009-10-09, 21:32
On Fri, Oct 9, 2009 at 1:27 PM, Tsz Wo (Nicholas), Sze <
[EMAIL PROTECTED]> wrote:
> Hi St.Ack,
> > ... soft lease to 1 second ...
> You are right that you don't have to change soft lease. It is for append
> but not related to hflash.
I should not have to set it then? I can remove this 70 second pause in
middle of my test?
> > Do I have to do open as another user?
> This should not be necessary.
> Could you send me/post your test?
Sure, as long as you don't hold this ugly code against me ever after.
I checked in the code so you could try it:
Its the first test, testSync.
It starts out by copying whats down in the hdfs testReadWhileWriting. That
bit works fine.
Then comes the ugly stuff.
HLog is our write-ahead log wrapper. Internally it writes out to a
SequenceFile.Writer. The SequenceFile.Writer has been doctored using
reflection so the out datamember is non-private. A call to HLog.sync runs
the SequenceFile.Writer.sync -- which DOES NOT call sync on the backing
output stream -- and then it calls sync on the now accessible out stream
(Sorry its so ugly -- I'm trying to hack stuff up fast so all of hbase gets
access to this new facility). If I trace in the debugger, I can see that
the sync on the out data member goes down into hflush. Queued up edits are
flushed. It seems like it should be working.
Do I have to do some doctoring of the reader? (It doesn't seem so given that
the code at the head of this test works).
Thanks for taking a look Nicholas.
To run the test, you can do "ant clean jar test -Dtestcase=TestHLog".
(Let me know if you want an eclipse .project + .classpath so you can get it
up in an ide to run debugger).
> Nicholas Sze
> *From:* stack <[EMAIL PROTECTED]>
> *To:* [EMAIL PROTECTED]
> *Sent:* Fri, October 9, 2009 1:13:37 PM
> *Subject:* hflush not working for me?
> I'm putting together some unit tests up in our application that exercise
> hflush. I'm using minidfscluster and a jar made by building head of the
> 0.21 branch of hdfs (from about a minute ago).
> Code opens a file, writes a bunch of edits, invokes hflush (by calling sync
> on DFSDataOutputStream instance) and then, without closing the Writer, opens
> a Reader on same file. This Reader does not see any edits not to mind edits
> up to the sync invocation.
> I can trace the code and see how on hflush it sends the queued packets of
> I studied TestReadWhileWriting. I've set setBoolean("dfs.support.append",
> true) before minidfscluster spins up. I can't set soft lease to 1 second
> because not in same package so I just wait out the default minute. It
> doesn't seem to make a difference.
> Do I have to do open as another user?
> Thanks for any pointers,