Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Re: copyToLocal Failed inside the cleanup(.........) of Map task


Copy link to this message
-
Re: copyToLocal Failed inside the cleanup(.........) of Map task
Shahab Yunus 2013-06-03, 22:38
Check out Pages 217-220 of the Hadoop: The Definitive Guide book. It has
some nice explanation. Also,
http://whiteycode.blogspot.com/2012/06/hadoop-removing-empty-output-files.html

Plus as Jay said, explanation of your use-case might also be helpful.

Regards,
Shahab
On Mon, Jun 3, 2013 at 5:44 PM, samir das mohapatra <[EMAIL PROTECTED]
> wrote:

> Do you have any link or example ? could you please send me ?
>
>
> On Tue, Jun 4, 2013 at 2:53 AM, Shahab Yunus <[EMAIL PROTECTED]>wrote:
>
>> Have you taken a look into extending the FileOutputFormat class and
>> overriding the OutputCommitter API functionality?
>>
>> Regards,
>> Shahab
>>
>>
>> On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Dear All,
>>>
>>>      Is there any way to copy the intermediate output file of the mapper
>>> into     local  folder  after each map task complete.
>>>
>>>    Right now I am using
>>>
>>>    FileSystem.copyToLocalFile(hdfsLocation,localLocation);
>>>  indiste the cleanup of mapper task , but it is failing .
>>>
>>> Exception file not found.
>>>
>>> But if I am giving same statement after the job complete in driver class
>>> ,it is working fine. that i dont want.
>>>
>>> protected void cleanup(Context context){
>>>    FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
>>> }
>>>
>>> Note: I need to copy the inter mediate output of the mapper to local
>>> file system just after  each map task complete. I dont want any reducer.
>>>
>>>    If this is not the right solution then how to solve this type of
>>> scenario.
>>>
>>> Any help.............
>>>
>>> regards,
>>> samir.
>>>
>>
>>
>