Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> hflush not working for me?


Copy link to this message
-
Re: hflush not working for me?
On Fri, Oct 9, 2009 at 1:27 PM, Tsz Wo (Nicholas), Sze <
[EMAIL PROTECTED]> wrote:

> Hi St.Ack,
>
> > ... soft lease to 1 second ...
> You are right that you don't have to change soft lease.  It is for append
> but not related to hflash.
>

I should not have to set it then?  I can remove this 70 second pause in
middle of my test?

>
> > Do I have to do open as another user?
> This should not be necessary.
>
> Could you send me/post your test?
>
>
Sure, as long as you don't hold this ugly code against me ever after.

I checked in the code so you could try it:
http://svn.apache.org/repos/asf/hadoop/hbase/trunk/src/test/org/apache/hadoop/hbase/regionserver/TestHLog.java

Its the first test, testSync.

It starts out by copying whats down in the hdfs testReadWhileWriting.  That
bit works fine.

Then comes the ugly stuff.

HLog is our write-ahead log wrapper.  Internally it writes out to a
SequenceFile.Writer.  The SequenceFile.Writer has been doctored using
reflection so the out datamember is non-private.  A call to HLog.sync runs
the SequenceFile.Writer.sync -- which DOES NOT call sync on the backing
output stream -- and then it calls sync on the now accessible out stream
(Sorry its so ugly -- I'm trying to hack stuff up fast so all of hbase gets
access to this new facility).  If I trace in the debugger, I can see that
the sync on the out data member goes down into hflush.  Queued up edits are
flushed.  It seems like it should be working.

Do I have to do some doctoring of the reader? (It doesn't seem so given that
the code at the head of this test works).

Thanks for taking a look Nicholas.

To run the test, you can do "ant clean jar test -Dtestcase=TestHLog".

(Let me know if you want an eclipse .project + .classpath so you can get it
up in an ide to run debugger).

St.Ack

> Nicholas Sze
>
>
> *From:* stack <[EMAIL PROTECTED]>
> *To:* [EMAIL PROTECTED]
> *Sent:* Fri, October 9, 2009 1:13:37 PM
> *Subject:* hflush not working for me?
>
> I'm putting together some unit tests up in our application that exercise
> hflush.  I'm using minidfscluster and a jar made by building head of the
> 0.21 branch of hdfs (from about a minute ago).
>
> Code opens a file, writes a bunch of edits, invokes hflush (by calling sync
> on DFSDataOutputStream instance) and then, without closing the Writer, opens
> a Reader on same file.  This Reader does not see any edits not to mind edits
> up to the sync invocation.
>
> I can trace the code and see how on hflush it sends the queued packets of
> edits.
>
> I studied TestReadWhileWriting.  I've set setBoolean("dfs.support.append",
> true) before minidfscluster spins up.  I can't set soft lease to 1 second
> because not in same package so I just wait out the default minute.  It
> doesn't seem to make a difference.
>
> Do I have to do open as another user?
>
> Thanks for any pointers,
> St.Ack
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB