Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> setLocalResources() on ContainerLaunchContext


Copy link to this message
-
Re: setLocalResources() on ContainerLaunchContext
Hi Harsh,

  Please see if this is useful, I got a stack trace after the error has
occurred....

2013-08-06 00:55:30,559 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set
to
/tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004 file:/tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
2013-08-06 00:55:31,017 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
exist: hdfs://isredeng/kishore/kk.ksh
2013-08-06 00:55:31,029 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
DEBUG: FAILED { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null }, File does
not exist: hdfs://isredeng/kishore/kk.ksh
2013-08-06 00:55:31,031 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource hdfs://isredeng/kishore/kk.ksh transitioned from DOWNLOADING to
FAILED
2013-08-06 00:55:31,034 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1375716148174_0004_01_000002 transitioned from
LOCALIZING to LOCALIZATION_FAILED
2013-08-06 00:55:31,035 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl:
Container container_1375716148174_0004_01_000002 sent RELEASE event on a
resource request { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null } not
present in cache.
2013-08-06 00:55:31,036 WARN org.apache.hadoop.ipc.Client: interrupted
waiting to send rpc request to server
java.lang.InterruptedException
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1290)
        at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:229)
        at java.util.concurrent.FutureTask.get(FutureTask.java:94)
        at
org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:930)
        at org.apache.hadoop.ipc.Client.call(Client.java:1285)
        at org.apache.hadoop.ipc.Client.call(Client.java:1264)
        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at $Proxy22.heartbeat(Unknown Source)
        at
org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
        at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:249)
        at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:163)
        at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106)
        at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:979)

And here is my code snippet:

      ContainerLaunchContext ctx Records.newRecord(ContainerLaunchContext.class);

      ctx.setEnvironment(oshEnv);

      // Set the local resources
      Map<String, LocalResource> localResources = new HashMap<String,
LocalResource>();

      LocalResource shellRsrc = Records.newRecord(LocalResource.class);
      shellRsrc.setType(LocalResourceType.FILE);
      shellRsrc.setVisibility(LocalResourceVisibility.APPLICATION);
      String shellScriptPath = "hdfs://isredeng//kishore/kk.ksh";
      try {
        shellRsrc.setResource(ConverterUtils.getYarnUrlFromURI(new
URI(shellScriptPath)));
      } catch (URISyntaxException e) {
        LOG.error("Error when trying to use shell script path specified"
            + " in env, path=" + shellScriptPath);
        e.printStackTrace();
      }

      shellRsrc.setTimestamp(0/*shellScriptPathTimestamp*/);
      shellRsrc.setSize(0/*shellScriptPathLen*/);
      String ExecShellStringPath = "ExecShellScript.sh";
      localResources.put(ExecShellStringPath, shellRsrc);

      ctx.setLocalResources(localResources);
Please let me know if you need anything else.

Thanks,
Kishore

On Tue, Aug 6, 2013 at 12:05 AM, Harsh J <[EMAIL PROTECTED]> wrote: