Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - hflush not working for me?


Copy link to this message
-
Re: hflush not working for me?
stack 2009-10-09, 21:32
On Fri, Oct 9, 2009 at 1:27 PM, Tsz Wo (Nicholas), Sze <
[EMAIL PROTECTED]> wrote:

> Hi St.Ack,
>
> > ... soft lease to 1 second ...
> You are right that you don't have to change soft lease.  It is for append
> but not related to hflash.
>

I should not have to set it then?  I can remove this 70 second pause in
middle of my test?

>
> > Do I have to do open as another user?
> This should not be necessary.
>
> Could you send me/post your test?
>
>
Sure, as long as you don't hold this ugly code against me ever after.

I checked in the code so you could try it:
http://svn.apache.org/repos/asf/hadoop/hbase/trunk/src/test/org/apache/hadoop/hbase/regionserver/TestHLog.java

Its the first test, testSync.

It starts out by copying whats down in the hdfs testReadWhileWriting.  That
bit works fine.

Then comes the ugly stuff.

HLog is our write-ahead log wrapper.  Internally it writes out to a
SequenceFile.Writer.  The SequenceFile.Writer has been doctored using
reflection so the out datamember is non-private.  A call to HLog.sync runs
the SequenceFile.Writer.sync -- which DOES NOT call sync on the backing
output stream -- and then it calls sync on the now accessible out stream
(Sorry its so ugly -- I'm trying to hack stuff up fast so all of hbase gets
access to this new facility).  If I trace in the debugger, I can see that
the sync on the out data member goes down into hflush.  Queued up edits are
flushed.  It seems like it should be working.

Do I have to do some doctoring of the reader? (It doesn't seem so given that
the code at the head of this test works).

Thanks for taking a look Nicholas.

To run the test, you can do "ant clean jar test -Dtestcase=TestHLog".

(Let me know if you want an eclipse .project + .classpath so you can get it
up in an ide to run debugger).

St.Ack

> Nicholas Sze
>
>
> *From:* stack <[EMAIL PROTECTED]>
> *To:* [EMAIL PROTECTED]
> *Sent:* Fri, October 9, 2009 1:13:37 PM
> *Subject:* hflush not working for me?
>
> I'm putting together some unit tests up in our application that exercise
> hflush.  I'm using minidfscluster and a jar made by building head of the
> 0.21 branch of hdfs (from about a minute ago).
>
> Code opens a file, writes a bunch of edits, invokes hflush (by calling sync
> on DFSDataOutputStream instance) and then, without closing the Writer, opens
> a Reader on same file.  This Reader does not see any edits not to mind edits
> up to the sync invocation.
>
> I can trace the code and see how on hflush it sends the queued packets of
> edits.
>
> I studied TestReadWhileWriting.  I've set setBoolean("dfs.support.append",
> true) before minidfscluster spins up.  I can't set soft lease to 1 second
> because not in same package so I just wait out the default minute.  It
> doesn't seem to make a difference.
>
> Do I have to do open as another user?
>
> Thanks for any pointers,
> St.Ack
>
>