Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> setLocalResources() on ContainerLaunchContext


Copy link to this message
-
Re: setLocalResources() on ContainerLaunchContext
To be honest, I've never tried loading a HDFS file onto the
LocalResource this way. I usually just pass a local file and that
works just fine. There may be something in the URI transformation
possibly breaking a HDFS source, but try passing a local file - does
that fail too? The Shell example uses a local file.

On Tue, Aug 6, 2013 at 10:54 AM, Krishna Kishore Bonagiri
<[EMAIL PROTECTED]> wrote:
> Hi Harsh,
>
>   Please see if this is useful, I got a stack trace after the error has
> occurred....
>
> 2013-08-06 00:55:30,559 INFO
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set
> to /tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
> > file:/tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
> 2013-08-06 00:55:31,017 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
> exist: hdfs://isredeng/kishore/kk.ksh
> 2013-08-06 00:55:31,029 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> DEBUG: FAILED { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null }, File does
> not exist: hdfs://isredeng/kishore/kk.ksh
> 2013-08-06 00:55:31,031 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> Resource hdfs://isredeng/kishore/kk.ksh transitioned from DOWNLOADING to
> FAILED
> 2013-08-06 00:55:31,034 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1375716148174_0004_01_000002 transitioned from
> LOCALIZING to LOCALIZATION_FAILED
> 2013-08-06 00:55:31,035 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl:
> Container container_1375716148174_0004_01_000002 sent RELEASE event on a
> resource request { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null } not
> present in cache.
> 2013-08-06 00:55:31,036 WARN org.apache.hadoop.ipc.Client: interrupted
> waiting to send rpc request to server
> java.lang.InterruptedException
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1290)
>         at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:229)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:94)
>         at
> org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:930)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1285)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1264)
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>         at $Proxy22.heartbeat(Unknown Source)
>         at
> org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
>         at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:249)
>         at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:163)
>         at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106)
>         at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:979)
>
>
>
> And here is my code snippet:
>
>       ContainerLaunchContext ctx > Records.newRecord(ContainerLaunchContext.class);
>
>       ctx.setEnvironment(oshEnv);
>
>       // Set the local resources
>       Map<String, LocalResource> localResources = new HashMap<String,
> LocalResource>();
>
>       LocalResource shellRsrc = Records.newRecord(LocalResource.class);
>       shellRsrc.setType(LocalResourceType.FILE);
>       shellRsrc.setVisibility(LocalResourceVisibility.APPLICATION);

Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB