Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: YARN: LocalResources and file distribution


Copy link to this message
-
Re: YARN: LocalResources and file distribution
Hi Arun,

  I have copied a shell script to HDFS and trying to execute it on
containers. How do I specify my shell script PATH in setCommands() call
on ContainerLaunchContext? I am doing it this way

      String shellScriptPath "hdfs://isredeng:8020/user/kbonagir/KKDummy/list.ksh";
      commands.add(shellScriptPath);

But my container execution is failing saying that there is No such file or
directory!

org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash:
hdfs://isredeng:8020/user/kbonagir/KKDummy/list.ksh: No such file or
directory

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
        at org.apache.hadoop.util.Shell.run(Shell.java:379)
        at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)

I could see this file with "hadoop fs" command and also saw messages in
Node Manager's log saying that the resource is downloaded and localized.
So, how do I run the downloaded shell script on a container?

Thanks,
Kishore

On Tue, Dec 3, 2013 at 4:57 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

> Robert,
>
>  YARN, by default, will only download *resource* from a shared namespace
> (e.g. HDFS).
>
>  If /home/hadoop/robert/large_jar.jar is available on each node then you
> can specify path as file:///home/hadoop/robert/large_jar.jar and it should
> work.
>
>  Else, you'll need to copy /home/hadoop/robert/large_jar.jar to HDFS and
> then specify hdfs://host:port/path/to/large_jar.jar.
>
> hth,
> Arun
>
> On Dec 1, 2013, at 12:03 PM, Robert Metzger <[EMAIL PROTECTED]> wrote:
>
> Hello,
>
> I'm currently writing code to run my application using Yarn (Hadoop 2.2.0).
> I used this code as a skeleton:
> https://github.com/hortonworks/simple-yarn-app
>
> Everything works fine on my local machine or on a cluster with the shared
> directories, but when I want to access resources outside of commonly
> accessible locations, my application fails.
>
> I have my application in a large jar file, containing everything
> (Submission Client, Application Master, and Workers).
> The submission client registers the large jar file as a local resource for
> the Application master's context.
>
> In my understanding, Yarn takes care of transferring the client-local
> resources to the application master's container.
> This is also stated here:
> http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
>
> You can use the LocalResource to add resources to your application
>> request. This will cause YARN to distribute the resource to the
>> ApplicationMaster node.
>
>
> If I'm starting my jar from the dir "/home/hadoop/robert/large_jar.jar",
> I'll get the following error from the nodemanager (another node in the
> cluster):
>
> 2013-12-01 20:13:00,810 INFO
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>> Failed to download rsrc { { file:/home/hadoop/robert/large_jar.jar, ..
>>
>
> So it seems as this node tries to access the file from its local file
> system.
>
> Do I have to use another "protocol" for the file, something like "
> file://host:port/home/blabla" ?
>
> Is it true that Yarn is able to distribute files (not using hdfs
> obviously?) ?
>
>
> The distributedshell-example suggests that I have to use HDFS:
> https://github.com/apache/hadoop-common/blob/50f0de14e377091c308c3a74ed089a7e4a7f0bfe/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
>
>
> Sincerely,
> Robert
>
>
>
>
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that