Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> HA for hadoop-0.20.2


Copy link to this message
-
HA for hadoop-0.20.2
I want to implement HA function for hadoop-0.20.2.

I  use the StandbyNN instance to implement the function. there is one
thread to  read the edits file in share directory,example NFS, and apply
transaction log to StandbyNN's namespace, we can call the thread is ingest
thread.

And the StandbyNN instance also do checkpoint  every one hour. When do
checkpoint, that need to wait ingest thread to finish the reading and the
ingest thread exit, and then the checkpoint is finished, StandbyNN instance
create new ingest thread to read new edits file(edits.new file is renamed
to edits file).

 If the checkpoint is fail, example upload image file is fail, in the case
the ingest thread is not exist. I need to handle the exception. I think the
simple way is to create new ingest thread to read edits file, but there is
possible that the same edits file is readed more than once,  so same
transaction log will be applied to StandbyNN‘s namespace more than once.
I learn the hadoop-2.0 code,  there are   below code  in
FSEditLogLoader.loadEditRecords method.
if (op.hasTransactionId()) {
if (op.getTransactionId() > expectedTxId) {
MetaRecoveryContext.editLogLoaderPrompt("There appears " +
"to be a gap in the edit log. We expected txid " +
expectedTxId + ", but got txid " +
op.getTransactionId() + ".", recovery, "ignoring missing " +
" transaction IDs");
} *else if (op.getTransactionId() < expectedTxId) {
MetaRecoveryContext.editLogLoaderPrompt("There appears " +
"to be an out-of-order edit in the edit log. We " +
"expected txid " + expectedTxId + ", but got txid " +
op.getTransactionId() + ".", recovery,
"skipping the out-of-order edit");
continue;*
}
}

The method use transaction id to guarantee same transaction log is not
applied to namespace more than once.

But in hadoop-0.20.2, FSEditLog don't store the transaction id into edits
log file. So I want to know if StandbyNN apply same transaction log to
namespace more than once, that will lead to the namespace of StandbyNN is
corrupt?
Please give me some advice.

Best Regards
LiuLei
+
Todd Lipcon 2012-11-14, 17:14