Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # dev - Help with error


+
Ralph Castain 2012-04-09, 16:10
+
Robert Evans 2012-04-09, 17:40
+
Ralph Castain 2012-04-09, 17:45
+
Kihwal Lee 2012-04-09, 19:27
+
Ralph Castain 2012-04-09, 20:13
+
Ralph Castain 2012-04-09, 20:26
+
Ralph Castain 2012-04-09, 20:30
Copy link to this message
-
Re: Help with error
Kihwal Lee 2012-04-09, 20:45
The path, "file:/Users/rhc/yarnrun/13", indicates that your copy operation's destination was the local file system, instead of hdfs.  What is the value of "fs.default.name" set to in core-site.xml?

Kihwal
On 4/9/12 3:26 PM, "Ralph Castain" <[EMAIL PROTECTED]> wrote:

Finally managed to chase down the 0.23 API docs and get the FileStatus definition. No real joy here - I output the path and got:

code:               LOG.info("destination path " + destStatus.getPath());

2012-04-09 14:22:48,359 INFO  Hamster (Hamster.java:getApplication(265)) - destination path file:/Users/rhc/yarnrun/13

However, when I attempt to list it:

Ralphs-iMac:bin rhc$ ./hdfs dfs -ls /Users/rhc/yarnrun
2012-04-09 14:22:57.640 java[14292:1903] Unable to load realm info from SCDynamicStore
2012-04-09 14:22:57.686 java[14292:1903] Unable to load realm info from SCDynamicStore
ls: `/Users/rhc/yarnrun': No such file or directory

I've been unable to track down the "realm" warnings, so I don't know if that is pertinent or not. It appears the files are not getting copied across, though the location looks okay to my eyes.
On Apr 9, 2012, at 1:27 PM, Kihwal Lee wrote:

> It looks like the home directory does not exist but the copy went through.
> Can you try to LOG the key fields in destStatus including path? It might be ending up in an unexpected place.
>
> Kihwal
>
>
>
> On 4/9/12 12:45 PM, "Ralph Castain" <[EMAIL PROTECTED]> wrote:
>
> Hi Bobby
>
> On Apr 9, 2012, at 11:40 AM, Robert Evans wrote:
>
>> What do you mean by relocated some supporting files to HDFS?  How do you relocate them?  What API do you use?
>
> I use the LocalResource and FileSystem classes to do the relocation, per the Hadoop example:
>
>        // set local resources for the application master
>        // local files or archives as needed
>        // In this scenario, the jar file for the application master is part of the local resources
>        Map<String, LocalResource> localResources = new HashMap<String, LocalResource>();
>
>        LOG.info("Copy openmpi tarball from local filesystem and add to local environment");
>        // Copy the application master jar to the filesystem
>        // Create a local resource to point to the destination jar path
>        FileSystem fs;
>        FileStatus destStatus;
>        try {
>            fs = FileSystem.get(conf);
>            Path src = new Path(pathOMPItarball);
>            String pathSuffix = appName + "/" + appId.getId();
>            Path dst = new Path(fs.getHomeDirectory(), pathSuffix);
>            try {
>                fs.copyFromLocalFile(false, true, src, dst);
>                try {
>                    destStatus = fs.getFileStatus(dst);
>                    LocalResource amJarRsrc = Records.newRecord(LocalResource.class);
>
>                    // Set the type of resource - file or archive
>                    // archives are untarred at destination
>                    amJarRsrc.setType(LocalResourceType.ARCHIVE);
>                    // Set visibility of the resource
>                    // Setting to most private option
>                    amJarRsrc.setVisibility(LocalResourceVisibility.APPLICATION);
>                    // Set the resource to be copied over
>                    amJarRsrc.setResource(ConverterUtils.getYarnUrlFromPath(dst));
>                    // Set timestamp and length of file so that the framework
>                    // can do basic sanity checks for the local resource
>                    // after it has been copied over to ensure it is the same
>                    // resource the client intended to use with the application
>                    amJarRsrc.setTimestamp(destStatus.getModificationTime());
>                    amJarRsrc.setSize(destStatus.getLen());
>                    localResources.put("openmpi",  amJarRsrc);
>                } catch (Throwable t) {
>                    LOG.fatal("Error on file status", t);
>                    System.exit(1);
>                }
>            } catch (Throwable t) {
+
Ralph Castain 2012-04-09, 21:50
+
Ralph Castain 2012-04-09, 21:51
+
Daryn Sharp 2012-04-10, 13:03
+
Milind.Bhandarkar@... 2012-04-09, 18:01