Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - setLocalResources() on ContainerLaunchContext


+
Krishna Kishore Bonagiri 2013-08-05, 09:14
+
Harsh J 2013-08-05, 09:30
+
Krishna Kishore Bonagiri 2013-08-05, 13:06
Copy link to this message
-
Re: setLocalResources() on ContainerLaunchContext
Harsh J 2013-08-05, 18:35
The detail is insufficient to answer why. You should also have gotten
a trace after it, can you post that? If possible, also the relevant
snippets of code.

On Mon, Aug 5, 2013 at 6:36 PM, Krishna Kishore Bonagiri
<[EMAIL PROTECTED]> wrote:
> Hi Harsh,
>  Thanks for the quick and detailed reply, it really helps. I am trying to
> use it and getting this error in node manager's log:
>
> 2013-08-05 08:57:28,867 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
> exist: hdfs://isredeng/kishore/kk.ksh
>
>
> This file is there on the machine with name "isredeng", I could do ls for
> that file as below:
>
> -bash-4.1$ hadoop fs -ls kishore/kk.ksh
> 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r--   3 dsadm supergroup       1046 2013-08-05 08:48 kishore/kk.ksh
>
> Note: I am using a single node cluster
>
> Thanks,
> Kishore
>
>
>
>
> On Mon, Aug 5, 2013 at 3:00 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> The string for each LocalResource in the map can be anything that
>> serves as a common identifier name for your application. At execution
>> time, the passed resource filename will be aliased to the name you've
>> mapped it to, so that the application code need not track special
>> names. The behavior is very similar to how you can, in MR, define a
>> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar).
>>
>> For an example, checkout the DistributedShell app sources.
>>
>> Over [1], you can see we take a user provided file path to a shell
>> script. This can be named anything as it is user-supplied.
>> Onto [2], we define this as a local resource [2.1] and embed it with a
>> different name (the string you ask about) [2.2], as defined at [3] as
>> an application reference-able constant.
>> Note that in [4], we add to the Container arguments the aliased name
>> we mapped it to (i.e. [3]) and not the original filename we received
>> from the user. The resource is placed on the container with this name
>> instead, so thats what we choose to execute.
>>
>> [1] -
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390
>>
>> [2] - [2.1]
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764
>> and [2.2]
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780
>>
>> [3] -
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205
>>
>> [4] -
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791
>>
>> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri
>> <[EMAIL PROTECTED]> wrote:
>> > Hi,
>> >
>> >   Can someone please tell me what is the use of calling
>> > setLocalResources()
>> > on ContainerLaunchContext?
>> >
>> >   And, also an example of how to use this will help...
>> >
>> >  I couldn't guess what is the String in the map that is passed to
>> > setLocalResources() like below:
>> >
>> >       // Set the local resources
>> >       Map<String, LocalResource> localResources = new HashMap<String,

Harsh J
+
Krishna Kishore Bonagiri 2013-08-06, 05:24
+
Harsh J 2013-08-06, 10:06
+
Krishna Kishore Bonagiri 2013-08-06, 10:35
+
Harsh J 2013-08-06, 11:54
+
Harsh J 2013-08-07, 01:47
+
Harsh J 2013-08-07, 02:15
+
Omkar Joshi 2013-08-07, 18:19
+
Krishna Kishore Bonagiri 2013-08-07, 18:29