Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> setLocalResources() on ContainerLaunchContext


+
Krishna Kishore Bonagiri 2013-08-05, 09:14
+
Harsh J 2013-08-05, 09:30
+
Krishna Kishore Bonagiri 2013-08-05, 13:06
Copy link to this message
-
Re: setLocalResources() on ContainerLaunchContext
The detail is insufficient to answer why. You should also have gotten
a trace after it, can you post that? If possible, also the relevant
snippets of code.

On Mon, Aug 5, 2013 at 6:36 PM, Krishna Kishore Bonagiri
<[EMAIL PROTECTED]> wrote:
> Hi Harsh,
>  Thanks for the quick and detailed reply, it really helps. I am trying to
> use it and getting this error in node manager's log:
>
> 2013-08-05 08:57:28,867 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
> exist: hdfs://isredeng/kishore/kk.ksh
>
>
> This file is there on the machine with name "isredeng", I could do ls for
> that file as below:
>
> -bash-4.1$ hadoop fs -ls kishore/kk.ksh
> 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r--   3 dsadm supergroup       1046 2013-08-05 08:48 kishore/kk.ksh
>
> Note: I am using a single node cluster
>
> Thanks,
> Kishore
>
>
>
>
> On Mon, Aug 5, 2013 at 3:00 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> The string for each LocalResource in the map can be anything that
>> serves as a common identifier name for your application. At execution
>> time, the passed resource filename will be aliased to the name you've
>> mapped it to, so that the application code need not track special
>> names. The behavior is very similar to how you can, in MR, define a
>> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar).
>>
>> For an example, checkout the DistributedShell app sources.
>>
>> Over [1], you can see we take a user provided file path to a shell
>> script. This can be named anything as it is user-supplied.
>> Onto [2], we define this as a local resource [2.1] and embed it with a
>> different name (the string you ask about) [2.2], as defined at [3] as
>> an application reference-able constant.
>> Note that in [4], we add to the Container arguments the aliased name
>> we mapped it to (i.e. [3]) and not the original filename we received
>> from the user. The resource is placed on the container with this name
>> instead, so thats what we choose to execute.
>>
>> [1] -
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390
>>
>> [2] - [2.1]
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764
>> and [2.2]
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780
>>
>> [3] -
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205
>>
>> [4] -
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791
>>
>> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri
>> <[EMAIL PROTECTED]> wrote:
>> > Hi,
>> >
>> >   Can someone please tell me what is the use of calling
>> > setLocalResources()
>> > on ContainerLaunchContext?
>> >
>> >   And, also an example of how to use this will help...
>> >
>> >  I couldn't guess what is the String in the map that is passed to
>> > setLocalResources() like below:
>> >
>> >       // Set the local resources
>> >       Map<String, LocalResource> localResources = new HashMap<String,

Harsh J
+
Krishna Kishore Bonagiri 2013-08-06, 05:24
+
Harsh J 2013-08-06, 10:06
+
Krishna Kishore Bonagiri 2013-08-06, 10:35
+
Harsh J 2013-08-06, 11:54
+
Harsh J 2013-08-07, 01:47
+
Harsh J 2013-08-07, 02:15
+
Omkar Joshi 2013-08-07, 18:19
+
Krishna Kishore Bonagiri 2013-08-07, 18:29
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB