Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - One NameNode keeps Rolling Edit Log on HDFS Federation


+
YouPeng Yang 2013-02-24, 15:09
+
Harsh J 2013-02-24, 16:48
+
Michel Segel 2013-02-24, 17:53
+
Harsh J 2013-02-24, 19:46
Copy link to this message
-
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
YouPeng Yang 2013-02-25, 01:06
Hi Mike,Harsh

    Thank you for your reply.

     I need to time to digest the above stuff.

 Thanks

Regards
2013/2/25 Harsh J <[EMAIL PROTECTED]>

> Mike,
>
> I don't see how SPOF comes into the picture when HA is already present
> in the releases that also carry Federation and each NN (federated or
> non) can be assigned further Standby-NN roles. At this point we can
> stop using the word "SPOF" completely for HDFS. Would do great good
> for avoiding further FUD around this :)
>
> The federated NameNodes are namespace-divided; and typically I think
> users may use it to divide applications into using different
> namespaces each, while using the same set of compute and data nodes.
> So if HA isn't deployed, an application may be lost but not "portion
> of the cluster", since DNs will remain unaffected and so would the
> other namespaces.
>
> On Sun, Feb 24, 2013 at 11:23 PM, Michel Segel
> <[EMAIL PROTECTED]> wrote:
> > I think part of the confusion stems from the fact that federation of
> name nodes only splits the very large cluster in to smaller portions of the
> same cluster. If you lose a federated name node, you only lose a portion of
> the cluster not the whole thing. So now instead of one SPOF, you have two
> SPOFs.
> >
> > The advantage of Federation is that you reduce the amount of memory
> required for the NN.
> >
> >
> > Sent from a remote device. Please excuse any typos...
> >
> > Mike Segel
> >
> > On Feb 24, 2013, at 10:48 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> >
> >> Hi,
> >>
> >> Federated namenodes are independent of one another (except that they
> >> both get reports from all the/common DNs in the cluster). It is
> >> natural to see one roll its edit logs based on its own rate of
> >> metadata growth, as compared to the other. Their edits, image, etc.
> >> everything is independent - they also do not know about/talk to each
> >> other.
> >>
> >> Given the above, I'd say (1) is true and (2) is invalid, and (3) can
> >> be found in PDFs attached to
> >> https://issues.apache.org/jira/browse/HDFS-1052.
> >>
> >> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <
> [EMAIL PROTECTED]> wrote:
> >>> Hi All
> >>>
> >>> I'm testing the HDFS Federation.I have 2 namenodes in  my cluster.
> >>> I find that it is  Rolling Edit Log continuously on one
> namenode,however
> >>> the other one  changes nothing.
> >>>
> >>> My question :
> >>> 1. Is it the right situation.
> >>> 2. I have thought that the two namenodes shoud keep concurrency.
> >>> why they get differences.
> >>> 3. is there any advanced docs about HDFS Federation.
> >>>
> >>> thanks.
> >>>
> >>> Regards.
> >>
> >>
> >>
> >> --
> >> Harsh J
> >>
>
>
>
> --
> Harsh J
>
+
Azuryy Yu 2013-02-25, 02:40
+
YouPeng Yang 2013-02-25, 02:45
+
YouPeng Yang 2013-02-25, 04:46