Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Help with error


Copy link to this message
-
Re: Help with error
Solved the "realm" warning courtesy of stackoverflow:

export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"

solves it. Didn't help resolve the problem, as expected.

On Apr 9, 2012, at 2:26 PM, Ralph Castain wrote:

> Finally managed to chase down the 0.23 API docs and get the FileStatus definition. No real joy here - I output the path and got:
>
> code:    LOG.info("destination path " + destStatus.getPath());
>
> 2012-04-09 14:22:48,359 INFO  Hamster (Hamster.java:getApplication(265)) - destination path file:/Users/rhc/yarnrun/13
>
> However, when I attempt to list it:
>
> Ralphs-iMac:bin rhc$ ./hdfs dfs -ls /Users/rhc/yarnrun
> 2012-04-09 14:22:57.640 java[14292:1903] Unable to load realm info from SCDynamicStore
> 2012-04-09 14:22:57.686 java[14292:1903] Unable to load realm info from SCDynamicStore
> ls: `/Users/rhc/yarnrun': No such file or directory
>
> I've been unable to track down the "realm" warnings, so I don't know if that is pertinent or not. It appears the files are not getting copied across, though the location looks okay to my eyes.
>
>
> On Apr 9, 2012, at 1:27 PM, Kihwal Lee wrote:
>
>> It looks like the home directory does not exist but the copy went through.
>> Can you try to LOG the key fields in destStatus including path? It might be ending up in an unexpected place.
>>
>> Kihwal
>>
>>
>>
>> On 4/9/12 12:45 PM, "Ralph Castain" <[EMAIL PROTECTED]> wrote:
>>
>> Hi Bobby
>>
>> On Apr 9, 2012, at 11:40 AM, Robert Evans wrote:
>>
>>> What do you mean by relocated some supporting files to HDFS?  How do you relocate them?  What API do you use?
>>
>> I use the LocalResource and FileSystem classes to do the relocation, per the Hadoop example:
>>
>>       // set local resources for the application master
>>       // local files or archives as needed
>>       // In this scenario, the jar file for the application master is part of the local resources
>>       Map<String, LocalResource> localResources = new HashMap<String, LocalResource>();
>>
>>       LOG.info("Copy openmpi tarball from local filesystem and add to local environment");
>>       // Copy the application master jar to the filesystem
>>       // Create a local resource to point to the destination jar path
>>       FileSystem fs;
>>       FileStatus destStatus;
>>       try {
>>           fs = FileSystem.get(conf);
>>           Path src = new Path(pathOMPItarball);
>>           String pathSuffix = appName + "/" + appId.getId();
>>           Path dst = new Path(fs.getHomeDirectory(), pathSuffix);
>>           try {
>>               fs.copyFromLocalFile(false, true, src, dst);
>>               try {
>>                   destStatus = fs.getFileStatus(dst);
>>                   LocalResource amJarRsrc = Records.newRecord(LocalResource.class);
>>
>>                   // Set the type of resource - file or archive
>>                   // archives are untarred at destination
>>                   amJarRsrc.setType(LocalResourceType.ARCHIVE);
>>                   // Set visibility of the resource
>>                   // Setting to most private option
>>                   amJarRsrc.setVisibility(LocalResourceVisibility.APPLICATION);
>>                   // Set the resource to be copied over
>>                   amJarRsrc.setResource(ConverterUtils.getYarnUrlFromPath(dst));
>>                   // Set timestamp and length of file so that the framework
>>                   // can do basic sanity checks for the local resource
>>                   // after it has been copied over to ensure it is the same
>>                   // resource the client intended to use with the application
>>                   amJarRsrc.setTimestamp(destStatus.getModificationTime());
>>                   amJarRsrc.setSize(destStatus.getLen());
>>                   localResources.put("openmpi",  amJarRsrc);
>>               } catch (Throwable t) {
>>                   LOG.fatal("Error on file status", t);
>>                   System.exit(1);