Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - setLocalResources() on ContainerLaunchContext


Copy link to this message
-
Re: setLocalResources() on ContainerLaunchContext
Harsh J 2013-08-06, 11:54
Can you try passing a fully qualified local path? That is, including the
file:/ scheme
On Aug 6, 2013 4:05 PM, "Krishna Kishore Bonagiri" <[EMAIL PROTECTED]>
wrote:

> Hi Harsh,
>    The setResource() call on LocalResource() is expecting an argument of
> type org.apache.hadoop.yarn.api.records.URL which is converted from a
> string in the form of URI. This happens in the following call of
> Distributed Shell example,
>
> shellRsrc.setResource(ConverterUtils.getYarnUrlFromURI(new URI(
> shellScriptPath)));
>
> So, if I give a local file I get a parsing error like below, which is when
> I changed it to an HDFS file thinking that it should be given like that
> only. Could you please give an example of how else it could be used, using
> a local file as you are saying?
>
> 2013-08-06 06:23:12,942 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Failed to parse resource-request
> java.net.URISyntaxException: Expected scheme name at index 0:
> :///home_/dsadm/kishore/kk.ksh
>         at java.net.URI$Parser.fail(URI.java:2820)
>         at java.net.URI$Parser.failExpecting(URI.java:2826)
>         at java.net.URI$Parser.parse(URI.java:3015)
>         at java.net.URI.<init>(URI.java:747)
>         at
> org.apache.hadoop.yarn.util.ConverterUtils.getPathFromYarnURL(ConverterUtils.java:77)
>         at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourceRequest.<init>(LocalResourceRequest.java:46)
>
>
>
> On Tue, Aug 6, 2013 at 3:36 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> To be honest, I've never tried loading a HDFS file onto the
>> LocalResource this way. I usually just pass a local file and that
>> works just fine. There may be something in the URI transformation
>> possibly breaking a HDFS source, but try passing a local file - does
>> that fail too? The Shell example uses a local file.
>>
>> On Tue, Aug 6, 2013 at 10:54 AM, Krishna Kishore Bonagiri
>> <[EMAIL PROTECTED]> wrote:
>> > Hi Harsh,
>> >
>> >   Please see if this is useful, I got a stack trace after the error has
>> > occurred....
>> >
>> > 2013-08-06 00:55:30,559 INFO
>> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD
>> set
>> > to
>> /tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
>> > >> >
>> file:/tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
>> > 2013-08-06 00:55:31,017 ERROR
>> > org.apache.hadoop.security.UserGroupInformation:
>> PriviledgedActionException
>> > as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does
>> not
>> > exist: hdfs://isredeng/kishore/kk.ksh
>> > 2013-08-06 00:55:31,029 INFO
>> >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> > DEBUG: FAILED { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null }, File
>> does
>> > not exist: hdfs://isredeng/kishore/kk.ksh
>> > 2013-08-06 00:55:31,031 INFO
>> >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
>> > Resource hdfs://isredeng/kishore/kk.ksh transitioned from DOWNLOADING to
>> > FAILED
>> > 2013-08-06 00:55:31,034 INFO
>> >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
>> > Container container_1375716148174_0004_01_000002 transitioned from
>> > LOCALIZING to LOCALIZATION_FAILED
>> > 2013-08-06 00:55:31,035 INFO
>> >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl:
>> > Container container_1375716148174_0004_01_000002 sent RELEASE event on a
>> > resource request { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null } not
>> > present in cache.
>> > 2013-08-06 00:55:31,036 WARN org.apache.hadoop.ipc.Client: interrupted
>> > waiting to send rpc request to server
>> > java.lang.InterruptedException
>> >         at
>> >
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1290)
>> >         at