I am using hadoop0.20.2, now I want to use HDFS HA function.  I research
AvatarNode. I find if the StandbyNN do checkpoint fail, when next time the
StandbyNN do checkpoint, the same edits file is loaded again.  Can same
edits file be loaded more than once in hadoop0.20.2?
if not, what is the harm?

Thanks,

LiuLei

I research FSEditLog.loadFSEdits method, I find OP_ADD and OP_CLOSE
opration first  delete inode(call  FSDirectory.unprotectedDelete method),
then re-add inode(call FSDirectory.unprotectedAddFile method),  so if NN
read one OP_ADD transaction log and add one inode to namespace  for the
operation, when next time NN read the same transaction log form same edits
file,NN first delete the inode , then add same inode to namespace again,
So  I think this is no harmful,even if the same transaction log is loaded
more than once.
If my understand is wrong, welcom everyone to point it out.
Thanks,

LiuLei

2012/11/4 lei liu <[EMAIL PROTECTED]>

> I am using hadoop0.20.2, now I want to use HDFS HA function.  I research
> AvatarNode. I find if the StandbyNN do checkpoint fail, when next time the
> StandbyNN do checkpoint, the same edits file is loaded again.  Can same
> edits file be loaded more than once in hadoop0.20.2?
> if not, what is the harm?
>
> Thanks,
>
> LiuLei
>
>

Hi,

If you want to learn more about HA in HDFS, here are some slides from
a talk that Aaron T. Meyers and Suresh Srinivas gave:
http://www.slideshare.net/hortonworks/nn-ha-hadoop-worldfinal-10173419

branch-2 and later contain HDFS HA.

cheers,
Colin
On Sun, Nov 4, 2012 at 1:06 AM, lei liu <[EMAIL PROTECTED]> wrote:
> I am using hadoop0.20.2, now I want to use HDFS HA function.  I research
> AvatarNode. I find if the StandbyNN do checkpoint fail, when next time the
> StandbyNN do checkpoint, the same edits file is loaded again.  Can same
> edits file be loaded more than once in hadoop0.20.2?
> if not, what is the harm?
>
> Thanks,
>
> LiuLei
>

Hi Coli,
Thank you very much for your reply.

I am researching the hadoop2.0 code, I find there are below code in
FSEditLogLoader.loadEditRecords method.
<code>
   if (op.hasTransactionId()) {
            if (op.getTransactionId() > expectedTxId) {
              MetaRecoveryContext.editLogLoaderPrompt("There appears " +
                  "to be a gap in the edit log.  We expected txid " +
                  expectedTxId + ", but got txid " +
                  op.getTransactionId() + ".", recovery, "ignoring missing
" +
                  " transaction IDs");
            } *else if (op.getTransactionId() < expectedTxId) {
              MetaRecoveryContext.**editLogLoaderPrompt("There appears " +
                  "to be an out-of-order edit in the edit log.  We " +
                  "expected txid " + expectedTxId + ", but got txid " +
                  op.getTransactionId() + ".", recovery,
                  "skipping the out-of-order edit");
              continue;*
            }
          }
</code>

The method use transaction id to  **guarantee same transaction log is not
applied to namespace more than once.

I am using hadoop-0.20.2, that don't store the transaction id to edits log
file.
So I want to know if  StandbyNN apply same edits log file many times to
namespace, that will lead to the namespace of StandbyNN is corrupt?
2012/11/8 Colin McCabe <[EMAIL PROTECTED]>

> Hi,
>
> If you want to learn more about HA in HDFS, here are some slides from
> a talk that Aaron T. Meyers and Suresh Srinivas gave:
> http://www.slideshare.net/hortonworks/nn-ha-hadoop-worldfinal-10173419
>
> branch-2 and later contain HDFS HA.
>
> cheers,
> Colin
>
>
> On Sun, Nov 4, 2012 at 1:06 AM, lei liu <[EMAIL PROTECTED]> wrote:
> > I am using hadoop0.20.2, now I want to use HDFS HA function.  I research
> > AvatarNode. I find if the StandbyNN do checkpoint fail, when next time
> the
> > StandbyNN do checkpoint, the same edits file is loaded again.  Can same
> > edits file be loaded more than once in hadoop0.20.2?
> > if not, what is the harm?
> >
> > Thanks,
> >
> > LiuLei
> >
>