Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> setLocalResources() on ContainerLaunchContext


Copy link to this message
-
Re: setLocalResources() on ContainerLaunchContext
To be honest, I've never tried loading a HDFS file onto the
LocalResource this way. I usually just pass a local file and that
works just fine. There may be something in the URI transformation
possibly breaking a HDFS source, but try passing a local file - does
that fail too? The Shell example uses a local file.

On Tue, Aug 6, 2013 at 10:54 AM, Krishna Kishore Bonagiri
<[EMAIL PROTECTED]> wrote:
> Hi Harsh,
>
>   Please see if this is useful, I got a stack trace after the error has
> occurred....
>
> 2013-08-06 00:55:30,559 INFO
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set
> to /tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
> > file:/tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
> 2013-08-06 00:55:31,017 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
> exist: hdfs://isredeng/kishore/kk.ksh
> 2013-08-06 00:55:31,029 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> DEBUG: FAILED { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null }, File does
> not exist: hdfs://isredeng/kishore/kk.ksh
> 2013-08-06 00:55:31,031 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource hdfs://isredeng/kishore/kk.ksh transitioned from DOWNLOADING to
> FAILED
> 2013-08-06 00:55:31,034 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1375716148174_0004_01_000002 transitioned from
> LOCALIZING to LOCALIZATION_FAILED
> 2013-08-06 00:55:31,035 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl:
> Container container_1375716148174_0004_01_000002 sent RELEASE event on a
> resource request { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null } not
> present in cache.
> 2013-08-06 00:55:31,036 WARN org.apache.hadoop.ipc.Client: interrupted
> waiting to send rpc request to server
> java.lang.InterruptedException
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1290)
>         at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:229)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:94)
>         at
> org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:930)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1285)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1264)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>         at $Proxy22.heartbeat(Unknown Source)
>         at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
>         at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:249)
>         at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:163)
>         at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106)
>         at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:979)
>
>
>
> And here is my code snippet:
>
>       ContainerLaunchContext ctx > Records.newRecord(ContainerLaunchContext.class);
>
>       ctx.setEnvironment(oshEnv);
>
>       // Set the local resources
>       Map<String, LocalResource> localResources = new HashMap<String,
> LocalResource>();
>
>       LocalResource shellRsrc = Records.newRecord(LocalResource.class);
>       shellRsrc.setType(LocalResourceType.FILE);
>       shellRsrc.setVisibility(LocalResourceVisibility.APPLICATION);

Harsh J