Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> One NameNode keeps Rolling Edit Log on HDFS Federation


Copy link to this message
-
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Hi Mike,Harsh

    Thank you for your reply.

     I need to time to digest the above stuff.

 Thanks

Regards
2013/2/25 Harsh J <[EMAIL PROTECTED]>

> Mike,
>
> I don't see how SPOF comes into the picture when HA is already present
> in the releases that also carry Federation and each NN (federated or
> non) can be assigned further Standby-NN roles. At this point we can
> stop using the word "SPOF" completely for HDFS. Would do great good
> for avoiding further FUD around this :)
>
> The federated NameNodes are namespace-divided; and typically I think
> users may use it to divide applications into using different
> namespaces each, while using the same set of compute and data nodes.
> So if HA isn't deployed, an application may be lost but not "portion
> of the cluster", since DNs will remain unaffected and so would the
> other namespaces.
>
> On Sun, Feb 24, 2013 at 11:23 PM, Michel Segel
> <[EMAIL PROTECTED]> wrote:
> > I think part of the confusion stems from the fact that federation of
> name nodes only splits the very large cluster in to smaller portions of the
> same cluster. If you lose a federated name node, you only lose a portion of
> the cluster not the whole thing. So now instead of one SPOF, you have two
> SPOFs.
> >
> > The advantage of Federation is that you reduce the amount of memory
> required for the NN.
> >
> >
> > Sent from a remote device. Please excuse any typos...
> >
> > Mike Segel
> >
> > On Feb 24, 2013, at 10:48 AM, Harsh J <[EMAIL PROTECTED]> wrote:
> >
> >> Hi,
> >>
> >> Federated namenodes are independent of one another (except that they
> >> both get reports from all the/common DNs in the cluster). It is
> >> natural to see one roll its edit logs based on its own rate of
> >> metadata growth, as compared to the other. Their edits, image, etc.
> >> everything is independent - they also do not know about/talk to each
> >> other.
> >>
> >> Given the above, I'd say (1) is true and (2) is invalid, and (3) can
> >> be found in PDFs attached to
> >> https://issues.apache.org/jira/browse/HDFS-1052.
> >>
> >> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <
> [EMAIL PROTECTED]> wrote:
> >>> Hi All
> >>>
> >>> I'm testing the HDFS Federation.I have 2 namenodes in  my cluster.
> >>> I find that it is  Rolling Edit Log continuously on one
> namenode,however
> >>> the other one  changes nothing.
> >>>
> >>> My question :
> >>> 1. Is it the right situation.
> >>> 2. I have thought that the two namenodes shoud keep concurrency.
> >>> why they get differences.
> >>> 3. is there any advanced docs about HDFS Federation.
> >>>
> >>> thanks.
> >>>
> >>> Regards.
> >>
> >>
> >>
> >> --
> >> Harsh J
> >>
>
>
>
> --
> Harsh J
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB