Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Re: copyToLocal Failed inside the cleanup(.........) of Map task


+
samir das mohapatra 2013-06-03, 21:44
Copy link to this message
-
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Check out Pages 217-220 of the Hadoop: The Definitive Guide book. It has
some nice explanation. Also,
http://whiteycode.blogspot.com/2012/06/hadoop-removing-empty-output-files.html

Plus as Jay said, explanation of your use-case might also be helpful.

Regards,
Shahab
On Mon, Jun 3, 2013 at 5:44 PM, samir das mohapatra <[EMAIL PROTECTED]
> wrote:

> Do you have any link or example ? could you please send me ?
>
>
> On Tue, Jun 4, 2013 at 2:53 AM, Shahab Yunus <[EMAIL PROTECTED]>wrote:
>
>> Have you taken a look into extending the FileOutputFormat class and
>> overriding the OutputCommitter API functionality?
>>
>> Regards,
>> Shahab
>>
>>
>> On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Dear All,
>>>
>>>      Is there any way to copy the intermediate output file of the mapper
>>> into     local  folder  after each map task complete.
>>>
>>>    Right now I am using
>>>
>>>    FileSystem.copyToLocalFile(hdfsLocation,localLocation);
>>>  indiste the cleanup of mapper task , but it is failing .
>>>
>>> Exception file not found.
>>>
>>> But if I am giving same statement after the job complete in driver class
>>> ,it is working fine. that i dont want.
>>>
>>> protected void cleanup(Context context){
>>>    FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
>>> }
>>>
>>> Note: I need to copy the inter mediate output of the mapper to local
>>> file system just after  each map task complete. I dont want any reducer.
>>>
>>>    If this is not the right solution then how to solve this type of
>>> scenario.
>>>
>>> Any help.............
>>>
>>> regards,
>>> samir.
>>>
>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB