Namenode HA (HDFS-1623) feature is now in trunk (yay!). It adds support for
namenode high availability with active and standby namenodes, with manual
failover. I propose merging this feature to 0.23 branch. Recently Nicholas
merged protocol buffers/wire compatibility changes to 0.23. Merging
Namenode HA into 0.23 will make it a significant release.

I plan to complete this by next week after running some nightly/regressions
tests. Early next week, I plan to attach a 0.23 merge patch to HDFS-1623
jira for people interested in reviewing the patch.

Please vote.

Regards,
Suresh

Awesome! +1!

Sent from my iPhone

On Mar 2, 2012, at 6:16 PM, Suresh Srinivas <[EMAIL PROTECTED]> wrote:

> Namenode HA (HDFS-1623) feature is now in trunk (yay!). It adds support for
> namenode high availability with active and standby namenodes, with manual
> failover. I propose merging this feature to 0.23 branch. Recently Nicholas
> merged protocol buffers/wire compatibility changes to 0.23. Merging
> Namenode HA into 0.23 will make it a significant release.
>
> I plan to complete this by next week after running some nightly/regressions
> tests. Early next week, I plan to attach a 0.23 merge patch to HDFS-1623
> jira for people interested in reviewing the patch.
>
> Please vote.
>
> Regards,
> Suresh

+1, this'll be great. The sooner the better. Please let me know if I can be
of any help.

Thanks a lot for volunteering to do this, Suresh.

--
Aaron T. Myers
Software Engineer, Cloudera

On Fri, Mar 2, 2012 at 6:15 PM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:

> Namenode HA (HDFS-1623) feature is now in trunk (yay!). It adds support for
> namenode high availability with active and standby namenodes, with manual
> failover. I propose merging this feature to 0.23 branch. Recently Nicholas
> merged protocol buffers/wire compatibility changes to 0.23. Merging
> Namenode HA into 0.23 will make it a significant release.
>
> I plan to complete this by next week after running some nightly/regressions
> tests. Early next week, I plan to attach a 0.23 merge patch to HDFS-1623
> jira for people interested in reviewing the patch.
>
> Please vote.
>
> Regards,
> Suresh
>

Hey Suresh,

+1  Sounds great. Thanks for volunteering!

You'll probably want to merge HDFS-1580,  HDFS-1765, HDFS-2158,
HDFS-2188, HDFS-2334, HDFS-2476, HDFS-2477, and HDFS-2495 to branch-23
first as these conflict and the patch will contain a bunch of non-HA
stuff.

Thanks,
Eli

On Fri, Mar 2, 2012 at 6:15 PM, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
> Namenode HA (HDFS-1623) feature is now in trunk (yay!). It adds support for
> namenode high availability with active and standby namenodes, with manual
> failover. I propose merging this feature to 0.23 branch. Recently Nicholas
> merged protocol buffers/wire compatibility changes to 0.23. Merging
> Namenode HA into 0.23 will make it a significant release.
>
> I plan to complete this by next week after running some nightly/regressions
> tests. Early next week, I plan to attach a 0.23 merge patch to HDFS-1623
> jira for people interested in reviewing the patch.
>
> Please vote.
>
> Regards,
> Suresh

Thanks Eli for the list. I have merged the bugs you identified to 0.23. I
am in the process of merging HA change. I have several conflicts, stemming
from the changes that are no in 0.23. Pushing following jiras to 0.23 will
make the merge easier. I believe these are mostly straight forward changes:

HADOOP-7557 Make IPC header be extensible
HDFS-3003. Remove getHostPortString() from NameNode, replace it with
NetUtils.getHostPortString().
HADOOP-8108. Move method getHostPortString() from NameNode to NetUtils.
HDFS-2764. TestBackupNode is racy.
HDFS-2430. The number of failed or low-resource volumes the NN can tolerate
should be configurable.
HADOOP-7358. Improve log levels when exceptions caught in RPC handler.
HDFS-2410. Further cleanup of hardcoded configuration keys and values.
HDFS-2285. BackupNode should reject requests to modify namespace.
HADOOP-7729. Send back valid HTTP response if user hits IPC port with HTTP
GET.
HADOOP-7717. Move handling of concurrent client fail-overs to
RetryInvocationHandler

If no one gets back to me in a day, I will start merging these to 0.23.

On Sun, Mar 4, 2012 at 9:43 AM, Eli Collins <[EMAIL PROTECTED]> wrote:

> Hey Suresh,
>
> +1  Sounds great. Thanks for volunteering!
>
> You'll probably want to merge HDFS-1580,  HDFS-1765, HDFS-2158,
> HDFS-2188, HDFS-2334, HDFS-2476, HDFS-2477, and HDFS-2495 to branch-23
> first as these conflict and the patch will contain a bunch of non-HA
> stuff.
>
> Thanks,
> Eli
>
> On Fri, Mar 2, 2012 at 6:15 PM, Suresh Srinivas <[EMAIL PROTECTED]>
> wrote:
> > Namenode HA (HDFS-1623) feature is now in trunk (yay!). It adds support
> for
> > namenode high availability with active and standby namenodes, with manual
> > failover. I propose merging this feature to 0.23 branch. Recently
> Nicholas
> > merged protocol buffers/wire compatibility changes to 0.23. Merging
> > Namenode HA into 0.23 will make it a significant release.
> >
> > I plan to complete this by next week after running some
> nightly/regressions
> > tests. Early next week, I plan to attach a 0.23 merge patch to HDFS-1623
> > jira for people interested in reviewing the patch.
> >
> > Please vote.
> >
> > Regards,
> > Suresh
>

+1 on merging all of those

On Wed, Mar 7, 2012 at 1:12 AM, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
> Thanks Eli for the list. I have merged the bugs you identified to 0.23. I
> am in the process of merging HA change. I have several conflicts, stemming
> from the changes that are no in 0.23. Pushing following jiras to 0.23 will
> make the merge easier. I believe these are mostly straight forward changes:
>
> HADOOP-7557 Make IPC header be extensible
> HDFS-3003. Remove getHostPortString() from NameNode, replace it with
> NetUtils.getHostPortString().
> HADOOP-8108. Move method getHostPortString() from NameNode to NetUtils.
> HDFS-2764. TestBackupNode is racy.
> HDFS-2430. The number of failed or low-resource volumes the NN can tolerate
> should be configurable.
> HADOOP-7358. Improve log levels when exceptions caught in RPC handler.
> HDFS-2410. Further cleanup of hardcoded configuration keys and values.
> HDFS-2285. BackupNode should reject requests to modify namespace.
> HADOOP-7729. Send back valid HTTP response if user hits IPC port with HTTP
> GET.
> HADOOP-7717. Move handling of concurrent client fail-overs to
> RetryInvocationHandler
>
> If no one gets back to me in a day, I will start merging these to 0.23.
>
> On Sun, Mar 4, 2012 at 9:43 AM, Eli Collins <[EMAIL PROTECTED]> wrote:
>
>> Hey Suresh,
>>
>> +1  Sounds great. Thanks for volunteering!
>>
>> You'll probably want to merge HDFS-1580,  HDFS-1765, HDFS-2158,
>> HDFS-2188, HDFS-2334, HDFS-2476, HDFS-2477, and HDFS-2495 to branch-23
>> first as these conflict and the patch will contain a bunch of non-HA
>> stuff.
>>
>> Thanks,
>> Eli
>>
>> On Fri, Mar 2, 2012 at 6:15 PM, Suresh Srinivas <[EMAIL PROTECTED]>
>> wrote:
>> > Namenode HA (HDFS-1623) feature is now in trunk (yay!). It adds support
>> for
>> > namenode high availability with active and standby namenodes, with manual
>> > failover. I propose merging this feature to 0.23 branch. Recently
>> Nicholas
>> > merged protocol buffers/wire compatibility changes to 0.23. Merging
>> > Namenode HA into 0.23 will make it a significant release.
>> >
>> > I plan to complete this by next week after running some
>> nightly/regressions
>> > tests. Early next week, I plan to attach a 0.23 merge patch to HDFS-1623
>> > jira for people interested in reviewing the patch.
>> >
>> > Please vote.
>> >
>> > Regards,
>> > Suresh
>>

--
Todd Lipcon
Software Engineer, Cloudera

Likewise, +1 to merging all of those. HADOOP-7717 and HDFS-2430 are hard
requirements for the HA work, and the rest of those seem like strict
improvements to me. If they make the merge easier, then by all means do it.

I don't think you need to wait a day, either. I doubt anyone will object to
merging these to 0.23.

Thanks a lot, Suresh.

--
Aaron T. Myers
Software Engineer, Cloudera

On Wed, Mar 7, 2012 at 1:12 AM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:

> Thanks Eli for the list. I have merged the bugs you identified to 0.23. I
> am in the process of merging HA change. I have several conflicts, stemming
> from the changes that are no in 0.23. Pushing following jiras to 0.23 will
> make the merge easier. I believe these are mostly straight forward changes:
>
> HADOOP-7557 Make IPC header be extensible
> HDFS-3003. Remove getHostPortString() from NameNode, replace it with
> NetUtils.getHostPortString().
> HADOOP-8108. Move method getHostPortString() from NameNode to NetUtils.
> HDFS-2764. TestBackupNode is racy.
> HDFS-2430. The number of failed or low-resource volumes the NN can tolerate
> should be configurable.
> HADOOP-7358. Improve log levels when exceptions caught in RPC handler.
> HDFS-2410. Further cleanup of hardcoded configuration keys and values.
> HDFS-2285. BackupNode should reject requests to modify namespace.
> HADOOP-7729. Send back valid HTTP response if user hits IPC port with HTTP
> GET.
> HADOOP-7717. Move handling of concurrent client fail-overs to
> RetryInvocationHandler
>
> If no one gets back to me in a day, I will start merging these to 0.23.
>
> On Sun, Mar 4, 2012 at 9:43 AM, Eli Collins <[EMAIL PROTECTED]> wrote:
>
> > Hey Suresh,
> >
> > +1  Sounds great. Thanks for volunteering!
> >
> > You'll probably want to merge HDFS-1580,  HDFS-1765, HDFS-2158,
> > HDFS-2188, HDFS-2334, HDFS-2476, HDFS-2477, and HDFS-2495 to branch-23
> > first as these conflict and the patch will contain a bunch of non-HA
> > stuff.
> >
> > Thanks,
> > Eli
> >
> > On Fri, Mar 2, 2012 at 6:15 PM, Suresh Srinivas <[EMAIL PROTECTED]>
> > wrote:
> > > Namenode HA (HDFS-1623) feature is now in trunk (yay!). It adds support
> > for
> > > namenode high availability with active and standby namenodes, with
> manual
> > > failover. I propose merging this feature to 0.23 branch. Recently
> > Nicholas
> > > merged protocol buffers/wire compatibility changes to 0.23. Merging
> > > Namenode HA into 0.23 will make it a significant release.
> > >
> > > I plan to complete this by next week after running some
> > nightly/regressions
> > > tests. Early next week, I plan to attach a 0.23 merge patch to
> HDFS-1623
> > > jira for people interested in reviewing the patch.
> > >
> > > Please vote.
> > >
> > > Regards,
> > > Suresh
> >
>

+1.  Thanks Suresh, those lgtm
On Wednesday, March 7, 2012, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
> Thanks Eli for the list. I have merged the bugs you identified to 0.23. I
> am in the process of merging HA change. I have several conflicts, stemming
> from the changes that are no in 0.23. Pushing following jiras to 0.23 will
> make the merge easier. I believe these are mostly straight forward
changes:
>
> HADOOP-7557 Make IPC header be extensible
> HDFS-3003. Remove getHostPortString() from NameNode, replace it with
> NetUtils.getHostPortString().
> HADOOP-8108. Move method getHostPortString() from NameNode to NetUtils.
> HDFS-2764. TestBackupNode is racy.
> HDFS-2430. The number of failed or low-resource volumes the NN can
tolerate
> should be configurable.
> HADOOP-7358. Improve log levels when exceptions caught in RPC handler.
> HDFS-2410. Further cleanup of hardcoded configuration keys and values.
> HDFS-2285. BackupNode should reject requests to modify namespace.
> HADOOP-7729. Send back valid HTTP response if user hits IPC port with HTTP
> GET.
> HADOOP-7717. Move handling of concurrent client fail-overs to
> RetryInvocationHandler
>
> If no one gets back to me in a day, I will start merging these to 0.23.
>
> On Sun, Mar 4, 2012 at 9:43 AM, Eli Collins <[EMAIL PROTECTED]> wrote:
>
>> Hey Suresh,
>>
>> +1  Sounds great. Thanks for volunteering!
>>
>> You'll probably want to merge HDFS-1580,  HDFS-1765, HDFS-2158,
>> HDFS-2188, HDFS-2334, HDFS-2476, HDFS-2477, and HDFS-2495 to branch-23
>> first as these conflict and the patch will contain a bunch of non-HA
>> stuff.
>>
>> Thanks,
>> Eli
>>
>> On Fri, Mar 2, 2012 at 6:15 PM, Suresh Srinivas <[EMAIL PROTECTED]>
>> wrote:
>> > Namenode HA (HDFS-1623) feature is now in trunk (yay!). It adds support
>> for
>> > namenode high availability with active and standby namenodes, with
manual
>> > failover. I propose merging this feature to 0.23 branch. Recently
>> Nicholas
>> > merged protocol buffers/wire compatibility changes to 0.23. Merging
>> > Namenode HA into 0.23 will make it a significant release.
>> >
>> > I plan to complete this by next week after running some
>> nightly/regressions
>> > tests. Early next week, I plan to attach a 0.23 merge patch to
HDFS-1623
>> > jira for people interested in reviewing the patch.
>> >
>> > Please vote.
>> >
>> > Regards,
>> > Suresh
>>
>

I have merged the change required for merging Namenode HA. I have also
attached a release 23 patch in the jira HDFS-1623. Please take a look the
attached patch and let me know if that looks good.

Regards,
Suresh

+1

I applied the patch to branch-0.23. It compiles just fine. I built a
distribution tar, deployed it to a 4-node cluster, and ran some smoke tests
with HA enabled. All seemed good.

I also ran the following unit tests, which should exercise the relevant HA
code:

TestOfflineEditsViewer,TestHDFSConcat,TestEditLogRace,TestNameEditsConfigs,TestSaveNamespace,TestEditLogFileOutputStream,TestFileJournalManager,TestCheckpoint,TestEditLog,TestFSEditLogLoader,TestFsLimits,TestSecurityTokenEditLog,TestStorageRestore,TestBackupNode,TestEditLogJournalFailures,TestEditLogTailer,TestEditLogsDuringFailover,TestFailureToReadEdits,TestHASafeMode,TestHAStateTransitions,TestFailureOfSharedDir,TestDNFencing,TestStandbyIsHot,TestGenericJournalConf,TestCheckPointForSecurityTokens,TestNNStorageRetentionManager,TestPersistBlocks,TestPBHelper,TestNNLeaseRecovery

Of these, they all passed except for TestOfflineEditsViewer and
TestPersistBlocks. These failed because the patch obviously doesn't include
changes to a few binary files which the tests rely on. Assuming that when
you merge to branch-0.23 you do an actual svn merge, and don't just apply
the patch, then these won't be a problem.

--
Aaron T. Myers
Software Engineer, Cloudera

On Wed, Mar 7, 2012 at 9:26 PM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:

> I have merged the change required for merging Namenode HA. I have also
> attached a release 23 patch in the jira HDFS-1623. Please take a look the
> attached patch and let me know if that looks good.
>
> Regards,
> Suresh
>

Hey Suresh,

Went through the patch, looks good to me. Put a +1 on jira.

Thanks,
Eli

On Wed, Mar 7, 2012 at 9:26 PM, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
> I have merged the change required for merging Namenode HA. I have also
> attached a release 23 patch in the jira HDFS-1623. Please take a look the
> attached patch and let me know if that looks good.
>
> Regards,
> Suresh

Hey Suresh,

Forgot to ask, when do you plan to check this into branch-23?  Will
delay any merges that might conflict with this until you've committed
it.

Thanks,
Eli

On Fri, Mar 9, 2012 at 10:08 AM, Eli Collins <[EMAIL PROTECTED]> wrote:
> Hey Suresh,
>
> Went through the patch, looks good to me. Put a +1 on jira.
>
> Thanks,
> Eli
>
> On Wed, Mar 7, 2012 at 9:26 PM, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
>> I have merged the change required for merging Namenode HA. I have also
>> attached a release 23 patch in the jira HDFS-1623. Please take a look the
>> attached patch and let me know if that looks good.
>>
>> Regards,
>> Suresh