Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Hlog Group Commit Question: SequenceFileLogReader

Copy link to this message
Hlog Group Commit Question: SequenceFileLogReader

I am trying to backport the HLog group commit functionality to Hbase 0.20.  For proper reliability, I am working with Dhruba to get the 0.21 syncFs() changes from HDFS ported back to HDFS 0.20 as well.  When going through a peer review of the modified code, my group had a question about the SequenceFileLogReader.java (WALReader).  I am hoping that you guys could be of assistance.

I know that there is an open issue [HBASE-2069] where Hlog::splitLog() does not call DFSDataInputStream::getVisibleLength(), which would properly sync hflushed, but unclosed, file lengths.  I believe the current workaround is to open an HDFS file in append mode & then close, which would cause the namenode to get updates from the datanodes.  However, I don’t see that shim present in Hlog::splitLog() on the 0.21 trunk.  Is this a pending issue to fix or is calling FSDataInputStream::available() within WALReaderFsDataInputStream::getPos() sufficient to force the namenode to sync up with the datanodes?

Nicolas Spiegelberg