Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> setLocalResources() on ContainerLaunchContext


+
Krishna Kishore Bonagiri 2013-08-05, 09:14
+
Harsh J 2013-08-05, 09:30
Copy link to this message
-
Re: setLocalResources() on ContainerLaunchContext
Hi Harsh,
 Thanks for the quick and detailed reply, it really helps. I am trying to
use it and getting this error in node manager's log:

2013-08-05 08:57:28,867 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
exist: hdfs://isredeng/kishore/kk.ksh
This file is there on the machine with name "isredeng", I could do ls for
that file as below:

-bash-4.1$ hadoop fs -ls kishore/kk.ksh
13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r--   3 dsadm supergroup       1046 2013-08-05 08:48 kishore/kk.ksh

Note: I am using a single node cluster

Thanks,
Kishore
On Mon, Aug 5, 2013 at 3:00 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> The string for each LocalResource in the map can be anything that
> serves as a common identifier name for your application. At execution
> time, the passed resource filename will be aliased to the name you've
> mapped it to, so that the application code need not track special
> names. The behavior is very similar to how you can, in MR, define a
> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar).
>
> For an example, checkout the DistributedShell app sources.
>
> Over [1], you can see we take a user provided file path to a shell
> script. This can be named anything as it is user-supplied.
> Onto [2], we define this as a local resource [2.1] and embed it with a
> different name (the string you ask about) [2.2], as defined at [3] as
> an application reference-able constant.
> Note that in [4], we add to the Container arguments the aliased name
> we mapped it to (i.e. [3]) and not the original filename we received
> from the user. The resource is placed on the container with this name
> instead, so thats what we choose to execute.
>
> [1] -
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390
>
> [2] - [2.1]
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764
> and [2.2]
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780
>
> [3] -
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205
>
> [4] -
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791
>
> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> >   Can someone please tell me what is the use of calling
> setLocalResources()
> > on ContainerLaunchContext?
> >
> >   And, also an example of how to use this will help...
> >
> >  I couldn't guess what is the String in the map that is passed to
> > setLocalResources() like below:
> >
> >       // Set the local resources
> >       Map<String, LocalResource> localResources = new HashMap<String,
> > LocalResource>();
> >
> > Thanks,
> > Kishore
> >
>
>
>
> --
> Harsh J
>
+
Harsh J 2013-08-05, 18:35
+
Krishna Kishore Bonagiri 2013-08-06, 05:24
+
Harsh J 2013-08-06, 10:06
+
Krishna Kishore Bonagiri 2013-08-06, 10:35
+
Harsh J 2013-08-06, 11:54
+
Harsh J 2013-08-07, 01:47
+
Harsh J 2013-08-07, 02:15
+
Omkar Joshi 2013-08-07, 18:19
+
Krishna Kishore Bonagiri 2013-08-07, 18:29