Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Re: HDFS Backup nodes


Copy link to this message
-
Re: HDFS Backup nodes
Srivas,

As you may know already, NFS is just being used in the first prototype for
HA.

Two options for editlog store are:
1. Using BookKeeper. Work has already completed on trunk towards this. This
will replace need for NFS to  store the editlogs and is highly available.
This solution will also be used for HA.
2. We have a short term goal also to enable editlogs going to HDFS itself.
The work is in progress.

Regards,
Suresh
>
> ---------- Forwarded message ----------
> From: M. C. Srivas <[EMAIL PROTECTED]>
> Date: Sun, Dec 11, 2011 at 10:47 PM
> Subject: Re: HDFS Backup nodes
> To: [EMAIL PROTECTED]
>
>
> You are out of luck if you don't want to use NFS, and yet want redundancy
> for the NN.  Even the new "NN HA" work being done by the community will
> require NFS ... and the NFS itself needs to be HA.
>
> But if you use a Netapp, then the likelihood of the Netapp crashing is
> lower than the likelihood of a garbage-collection-of-death happening in the
> NN.
>
> [ disclaimer:  I don't work for Netapp, I work for MapR ]
>
>
> On Wed, Dec 7, 2011 at 4:30 PM, randy <[EMAIL PROTECTED]> wrote:
>
> > Thanks Joey. We've had enough problems with nfs (mainly under very high
> > load) that we thought it might be riskier to use it for the NN.
> >
> > randy
> >
> >
> > On 12/07/2011 06:46 PM, Joey Echeverria wrote:
> >
> >> Hey Rand,
> >>
> >> It will mark that storage directory as failed and ignore it from then
> >> on. In order to do this correctly, you need a couple of options
> >> enabled on the NFS mount to make sure that it doesn't retry
> >> infinitely. I usually run with the tcp,soft,intr,timeo=10,**retrans=10
> >> options set.
> >>
> >> -Joey
> >>
> >> On Wed, Dec 7, 2011 at 12:37 PM,<[EMAIL PROTECTED]>  wrote:
> >>
> >>> What happens then if the nfs server fails or isn't reachable? Does hdfs
> >>> lock up? Does it gracefully ignore the nfs copy?
> >>>
> >>> Thanks,
> >>> randy
> >>>
> >>> ----- Original Message -----
> >>> From: "Joey Echeverria"<[EMAIL PROTECTED]>
> >>> To: [EMAIL PROTECTED]
> >>> Sent: Wednesday, December 7, 2011 6:07:58 AM
> >>> Subject: Re: HDFS Backup nodes
> >>>
> >>> You should also configure the Namenode to use an NFS mount for one of
> >>> it's storage directories. That will give the most up-to-date back of
> >>> the metadata in case of total node failure.
> >>>
> >>> -Joey
> >>>
> >>> On Wed, Dec 7, 2011 at 3:17 AM, praveenesh kumar<[EMAIL PROTECTED]>
> >>>  wrote:
> >>>
> >>>> This means still we are relying on Secondary NameNode idealogy for
> >>>> Namenode's backup.
> >>>> Can OS-mirroring of Namenode is a good alternative keep it alive all
> the
> >>>> time ?
> >>>>
> >>>> Thanks,
> >>>> Praveenesh
> >>>>
> >>>> On Wed, Dec 7, 2011 at 1:35 PM, Uma Maheswara Rao G<
> >>>> [EMAIL PROTECTED]>wrote:
> >>>>
> >>>>  AFAIK backup node introduced in 0.21 version onwards.
> >>>>> ______________________________**__________
> >>>>> From: praveenesh kumar [[EMAIL PROTECTED]]
> >>>>> Sent: Wednesday, December 07, 2011 12:40 PM
> >>>>> To: [EMAIL PROTECTED]
> >>>>> Subject: HDFS Backup nodes
> >>>>>
> >>>>> Does hadoop 0.20.205 supports configuring HDFS backup nodes ?
> >>>>>
> >>>>> Thanks,
> >>>>> Praveenesh
> >>>>>
> >>>>>
> >>>
> >>>
> >>> --
> >>> Joseph Echeverria
> >>> Cloudera, Inc.
> >>> 443.305.9434
> >>>
> >>
> >>
> >>
> >>
> >
>
>
+
M. C. Srivas 2011-12-14, 07:00
+
Konstantin Boudnik 2011-12-14, 07:28
+
Scott Carey 2011-12-14, 18:09
+
Konstantin Boudnik 2011-12-14, 21:48
+
Todd Lipcon 2011-12-14, 07:39
+
Scott Carey 2011-12-14, 18:00
+
Todd Lipcon 2011-12-14, 21:57
+
praveenesh kumar 2011-12-07, 07:10
+
Uma Maheswara Rao G 2011-12-07, 08:05
+
praveenesh kumar 2011-12-07, 08:17
+
Sagar Shukla 2011-12-07, 08:23
+
Joey Echeverria 2011-12-07, 11:07
+
Jorn Argelo - Ephorus 2011-12-07, 13:25
+
Koji Noguchi 2011-12-07, 16:58
+
Jorn Argelo - Ephorus 2011-12-08, 08:26
+
randysch@... 2011-12-07, 17:37
+
Joey Echeverria 2011-12-07, 23:46
+
randy 2011-12-08, 00:30
+
M. C. Srivas 2011-12-12, 06:47
+
Todd Lipcon 2011-12-14, 02:19
+
M. C. Srivas 2011-12-14, 06:42
+
Todd Lipcon 2011-12-14, 07:25
+
Harsh J 2011-12-08, 01:32