-Re: Running Application (YARN) on top of federated NameNode
lohit 2012-10-29, 23:27
2012/10/29 Todd Lipcon <[EMAIL PROTECTED]>
> Hi Lohit,
> There are basically three main options here:
> 1) Symlinks. As you suggested, you could have one of the namespaces have
> top-levels cross-filesystem symlinks to the other explicit namespaces in
> your cluster. The downside of this is that currently symlinks are not well
> supported by the FileSystem API, so you may run into serious issues using
> it with MR applications.
> 2) Explicitly reference individual namespaces: this is basically separate
> HDFS clusters which share a pool of datanodes. If you are using namespaces
> to separate entirely separate applications, then the different apps would
> just reference their own namenodes with no knowledge that the storage
> underneath is pooled. Of course you may run a job which has input and
> output on different namesystems, and that's completely fine.
> 3) Use viewfs (client side mount tables). This is essentially a client-side
> mapping of viewfs paths to the other namenodes.
> Hope that helps
> On Mon, Oct 29, 2012 at 4:14 PM, lohit <[EMAIL PROTECTED]> wrote:
> > Hi Devs,
> > I am trying to understand about cluster setup with Federated NameNodes
> > YARN (or MR1) on top of it specifically.
> > From federation documentation (
> > )
> > I can see how each namenode will have its own namespace.
> > Can somebody help me understand how would YARN work on this. If I have 2
> > NameServices, how would YARN work with both of them. YARN or clients
> > look at fs.defaultFS (which point to one NameService) to resolve to DFS,
> > right? Is the setup something like YARN and others would connect to one
> > nameservices (call it top level name service) and admins would setup
> > symlinks from different nameservcies to this top level name service?
> > Thanks,
> > Lohit
> Todd Lipcon
> Software Engineer, Cloudera
Have a Nice Day!