Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Help with error


Copy link to this message
-
Re: Help with error
The copyFromLocalFile method has a void return, but internally is calling FileUtil methods that may return a boolean for success.  False appears to be returned if the source file cannot be deleted, or if the dest directory cannot be created.  The boolean result is ignored by copyFromLocalFile leading the caller to believe the copy was successful.

I'm not sure if this bug is aggravating your situation, so I'd try to manually create the dest dir and remove the src file.

Daryn
On Apr 9, 2012, at 4:51 PM, Ralph Castain wrote:

>
> On Apr 9, 2012, at 3:50 PM, Ralph Castain wrote:
>
>>
>> On Apr 9, 2012, at 2:45 PM, Kihwal Lee wrote:
>>
>>> The path, "file:/Users/rhc/yarnrun/13", indicates that your copy operation's destination was the local file system, instead of hdfs.
>>
>> Yeah, I realized that too after I sent the note. Sure enough - the files are there.
>
>
> Quick correction: the path exists, but as a file instead of a directory, and therefore the files to be moved there don't exist.
>
>
>>
>>> What is the value of "fs.default.name" set to in core-site.xml?
>>
>> <configuration>
>> <property>
>> <name>fs.default.name</name>
>> <value>hdfs://localhost:9000</value>
>> </property>
>> </configuration>
>>
>>
>>>
>>> Kihwal
>>>
>>>
>>> On 4/9/12 3:26 PM, "Ralph Castain" <[EMAIL PROTECTED]> wrote:
>>>
>>> Finally managed to chase down the 0.23 API docs and get the FileStatus definition. No real joy here - I output the path and got:
>>>
>>> code:               LOG.info("destination path " + destStatus.getPath());
>>>
>>> 2012-04-09 14:22:48,359 INFO  Hamster (Hamster.java:getApplication(265)) - destination path file:/Users/rhc/yarnrun/13
>>>
>>> However, when I attempt to list it:
>>>
>>> Ralphs-iMac:bin rhc$ ./hdfs dfs -ls /Users/rhc/yarnrun
>>> 2012-04-09 14:22:57.640 java[14292:1903] Unable to load realm info from SCDynamicStore
>>> 2012-04-09 14:22:57.686 java[14292:1903] Unable to load realm info from SCDynamicStore
>>> ls: `/Users/rhc/yarnrun': No such file or directory
>>>
>>> I've been unable to track down the "realm" warnings, so I don't know if that is pertinent or not. It appears the files are not getting copied across, though the location looks okay to my eyes.
>>>
>>>
>>> On Apr 9, 2012, at 1:27 PM, Kihwal Lee wrote:
>>>
>>>> It looks like the home directory does not exist but the copy went through.
>>>> Can you try to LOG the key fields in destStatus including path? It might be ending up in an unexpected place.
>>>>
>>>> Kihwal
>>>>
>>>>
>>>>
>>>> On 4/9/12 12:45 PM, "Ralph Castain" <[EMAIL PROTECTED]> wrote:
>>>>
>>>> Hi Bobby
>>>>
>>>> On Apr 9, 2012, at 11:40 AM, Robert Evans wrote:
>>>>
>>>>> What do you mean by relocated some supporting files to HDFS?  How do you relocate them?  What API do you use?
>>>>
>>>> I use the LocalResource and FileSystem classes to do the relocation, per the Hadoop example:
>>>>
>>>>     // set local resources for the application master
>>>>     // local files or archives as needed
>>>>     // In this scenario, the jar file for the application master is part of the local resources
>>>>     Map<String, LocalResource> localResources = new HashMap<String, LocalResource>();
>>>>
>>>>     LOG.info("Copy openmpi tarball from local filesystem and add to local environment");
>>>>     // Copy the application master jar to the filesystem
>>>>     // Create a local resource to point to the destination jar path
>>>>     FileSystem fs;
>>>>     FileStatus destStatus;
>>>>     try {
>>>>         fs = FileSystem.get(conf);
>>>>         Path src = new Path(pathOMPItarball);
>>>>         String pathSuffix = appName + "/" + appId.getId();
>>>>         Path dst = new Path(fs.getHomeDirectory(), pathSuffix);
>>>>         try {
>>>>             fs.copyFromLocalFile(false, true, src, dst);
>>>>             try {
>>>>                 destStatus = fs.getFileStatus(dst);
>>>>                 LocalResource amJarRsrc = Records.newRecord(LocalResource.class);
>>>>
>>>>                 // Set the type of resource - file or archive
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB