Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Does Hadoop 1.0.4 provide a durable sync for HBase-0.94.6?


Copy link to this message
-
Re: Does Hadoop 1.0.4 provide a durable sync for HBase-0.94.6?
On Tue, May 28, 2013 at 7:09 AM, jingguo yao <[EMAIL PROTECTED]> wrote:

> Section 2.1.3 says that Hadoop 1.0.4 works with HBase-0.94.x [1]. And
> Section 2.1.3.3 says that 1.0.4 has a working durable sync. But when I
> check the source code of DFSClient.DFSOutputStream's sync method, I
> finds the following javadoc:
>
>     /**
>      * All data is written out to datanodes. It is not guaranteed
>      * that data has been flushed to persistent store on the
>      * datanode. Block allocations are persisted on namenode.
>      */
>
> So it seems that sync does not support a durable sync. It contradicts
> with [1].
>
> Can anybody help me on this confusion? Thanks.

This issue is probably the best source for the state of sync in hbase (and
hdfs): https://issues.apache.org/jira/browse/HBASE-5954

In short, the refguide is misleading -- let me fix -- as 1.0.4 indeed has a
sync but it is just a sync to the memory of three datanodes, not a true
fsync out to disk.  The above cited issue is tracking issues that our Lars
and other have contributed to HDFS to add fsync support.

Yours,
St.Ack
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB