Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> setLocalResources() on ContainerLaunchContext


Copy link to this message
-
Re: setLocalResources() on ContainerLaunchContext
Hi Harsh,

  Please see if this is useful, I got a stack trace after the error has
occurred....

2013-08-06 00:55:30,559 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set
to
/tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004 file:/tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
2013-08-06 00:55:31,017 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
exist: hdfs://isredeng/kishore/kk.ksh
2013-08-06 00:55:31,029 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
DEBUG: FAILED { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null }, File does
not exist: hdfs://isredeng/kishore/kk.ksh
2013-08-06 00:55:31,031 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource hdfs://isredeng/kishore/kk.ksh transitioned from DOWNLOADING to
FAILED
2013-08-06 00:55:31,034 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1375716148174_0004_01_000002 transitioned from
LOCALIZING to LOCALIZATION_FAILED
2013-08-06 00:55:31,035 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl:
Container container_1375716148174_0004_01_000002 sent RELEASE event on a
resource request { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null } not
present in cache.
2013-08-06 00:55:31,036 WARN org.apache.hadoop.ipc.Client: interrupted
waiting to send rpc request to server
java.lang.InterruptedException
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1290)
        at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:229)
        at java.util.concurrent.FutureTask.get(FutureTask.java:94)
        at
org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:930)
        at org.apache.hadoop.ipc.Client.call(Client.java:1285)
        at org.apache.hadoop.ipc.Client.call(Client.java:1264)
        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at $Proxy22.heartbeat(Unknown Source)
        at
org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
        at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:249)
        at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:163)
        at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106)
        at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:979)

And here is my code snippet:

      ContainerLaunchContext ctx Records.newRecord(ContainerLaunchContext.class);

      ctx.setEnvironment(oshEnv);

      // Set the local resources
      Map<String, LocalResource> localResources = new HashMap<String,
LocalResource>();

      LocalResource shellRsrc = Records.newRecord(LocalResource.class);
      shellRsrc.setType(LocalResourceType.FILE);
      shellRsrc.setVisibility(LocalResourceVisibility.APPLICATION);
      String shellScriptPath = "hdfs://isredeng//kishore/kk.ksh";
      try {
        shellRsrc.setResource(ConverterUtils.getYarnUrlFromURI(new
URI(shellScriptPath)));
      } catch (URISyntaxException e) {
        LOG.error("Error when trying to use shell script path specified"
            + " in env, path=" + shellScriptPath);
        e.printStackTrace();
      }

      shellRsrc.setTimestamp(0/*shellScriptPathTimestamp*/);
      shellRsrc.setSize(0/*shellScriptPathLen*/);
      String ExecShellStringPath = "ExecShellScript.sh";
      localResources.put(ExecShellStringPath, shellRsrc);

      ctx.setLocalResources(localResources);
Please let me know if you need anything else.

Thanks,
Kishore

On Tue, Aug 6, 2013 at 12:05 AM, Harsh J <[EMAIL PROTECTED]> wrote:

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB