Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> Running Application (YARN) on top of federated NameNode


Copy link to this message
-
Re: Running Application (YARN) on top of federated NameNode
Thanks Todd

2012/10/29 Todd Lipcon <[EMAIL PROTECTED]>

> Hi Lohit,
>
> There are basically three main options here:
>
> 1) Symlinks. As you suggested, you could have one of the namespaces have
> top-levels cross-filesystem symlinks to the other explicit namespaces in
> your cluster. The downside of this is that currently symlinks are not well
> supported by the FileSystem API, so you may run into serious issues using
> it with MR applications.
>
> 2) Explicitly reference individual namespaces: this is basically separate
> HDFS clusters which share a pool of datanodes. If you are using namespaces
> to separate entirely separate applications, then the different apps would
> just reference their own namenodes with no knowledge that the storage
> underneath is pooled. Of course you may run a job which has input and
> output on different namesystems, and that's completely fine.
>
> 3) Use viewfs (client side mount tables). This is essentially a client-side
> mapping of viewfs paths to the other namenodes.
>
> Hope that helps
>
> -Todd
>
> On Mon, Oct 29, 2012 at 4:14 PM, lohit <[EMAIL PROTECTED]> wrote:
>
> > Hi Devs,
> >
> > I am trying to understand about cluster setup with Federated NameNodes
> and
> > YARN (or MR1) on top of it specifically.
> > From federation documentation (
> >
> >
> http://hadoop.apache.org/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html
> > )
> > I can see how each namenode will have its own namespace.
> > Can somebody help me understand how would YARN work on this. If I have 2
> > NameServices, how would YARN work with both of them. YARN or clients
> would
> > look at fs.defaultFS (which point to one NameService) to resolve to DFS,
> > right? Is the setup something like YARN and others would connect to one
> > nameservices (call it top level name service) and admins would setup
> > symlinks from different nameservcies to this top level name service?
> >
> > Thanks,
> > Lohit
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--
Have a Nice Day!
Lohit
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB