Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - setLocalResources() on ContainerLaunchContext


+
Krishna Kishore Bonagiri 2013-08-05, 09:14
+
Harsh J 2013-08-05, 09:30
+
Krishna Kishore Bonagiri 2013-08-05, 13:06
+
Harsh J 2013-08-05, 18:35
+
Krishna Kishore Bonagiri 2013-08-06, 05:24
+
Harsh J 2013-08-06, 10:06
Copy link to this message
-
Re: setLocalResources() on ContainerLaunchContext
Krishna Kishore Bonagiri 2013-08-06, 10:35
Hi Harsh,
   The setResource() call on LocalResource() is expecting an argument of
type org.apache.hadoop.yarn.api.records.URL which is converted from a
string in the form of URI. This happens in the following call of
Distributed Shell example,

shellRsrc.setResource(ConverterUtils.getYarnUrlFromURI(new URI(
shellScriptPath)));

So, if I give a local file I get a parsing error like below, which is when
I changed it to an HDFS file thinking that it should be given like that
only. Could you please give an example of how else it could be used, using
a local file as you are saying?

2013-08-06 06:23:12,942 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Failed to parse resource-request
java.net.URISyntaxException: Expected scheme name at index 0:
:///home_/dsadm/kishore/kk.ksh
        at java.net.URI$Parser.fail(URI.java:2820)
        at java.net.URI$Parser.failExpecting(URI.java:2826)
        at java.net.URI$Parser.parse(URI.java:3015)
        at java.net.URI.<init>(URI.java:747)
        at
org.apache.hadoop.yarn.util.ConverterUtils.getPathFromYarnURL(ConverterUtils.java:77)
        at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourceRequest.<init>(LocalResourceRequest.java:46)

On Tue, Aug 6, 2013 at 3:36 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> To be honest, I've never tried loading a HDFS file onto the
> LocalResource this way. I usually just pass a local file and that
> works just fine. There may be something in the URI transformation
> possibly breaking a HDFS source, but try passing a local file - does
> that fail too? The Shell example uses a local file.
>
> On Tue, Aug 6, 2013 at 10:54 AM, Krishna Kishore Bonagiri
> <[EMAIL PROTECTED]> wrote:
> > Hi Harsh,
> >
> >   Please see if this is useful, I got a stack trace after the error has
> > occurred....
> >
> > 2013-08-06 00:55:30,559 INFO
> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD
> set
> > to
> /tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
> > > >
> file:/tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
> > 2013-08-06 00:55:31,017 ERROR
> > org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException
> > as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
> > exist: hdfs://isredeng/kishore/kk.ksh
> > 2013-08-06 00:55:31,029 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> > DEBUG: FAILED { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null }, File
> does
> > not exist: hdfs://isredeng/kishore/kk.ksh
> > 2013-08-06 00:55:31,031 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> > Resource hdfs://isredeng/kishore/kk.ksh transitioned from DOWNLOADING to
> > FAILED
> > 2013-08-06 00:55:31,034 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> > Container container_1375716148174_0004_01_000002 transitioned from
> > LOCALIZING to LOCALIZATION_FAILED
> > 2013-08-06 00:55:31,035 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl:
> > Container container_1375716148174_0004_01_000002 sent RELEASE event on a
> > resource request { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null } not
> > present in cache.
> > 2013-08-06 00:55:31,036 WARN org.apache.hadoop.ipc.Client: interrupted
> > waiting to send rpc request to server
> > java.lang.InterruptedException
> >         at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1290)
> >         at
> > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:229)
> >         at java.util.concurrent.FutureTask.get(FutureTask.java:94)
> >         at
> > org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:930)
> >         at org.apache.hadoop.ipc.Client.call(Client.java:1285)
+
Harsh J 2013-08-06, 11:54
+
Harsh J 2013-08-07, 01:47
+
Harsh J 2013-08-07, 02:15
+
Omkar Joshi 2013-08-07, 18:19
+
Krishna Kishore Bonagiri 2013-08-07, 18:29