Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - Re: copyToLocal Failed inside the cleanup(.........) of Map task


Copy link to this message
-
Re: copyToLocal Failed inside the cleanup(.........) of Map task
samir das mohapatra 2013-06-03, 21:44
Do you have any link or example ? could you please send me ?
On Tue, Jun 4, 2013 at 2:53 AM, Shahab Yunus <[EMAIL PROTECTED]> wrote:

> Have you taken a look into extending the FileOutputFormat class and
> overriding the OutputCommitter API functionality?
>
> Regards,
> Shahab
>
>
> On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra <
> [EMAIL PROTECTED]> wrote:
>
>> Dear All,
>>
>>      Is there any way to copy the intermediate output file of the mapper
>> into     local  folder  after each map task complete.
>>
>>    Right now I am using
>>
>>    FileSystem.copyToLocalFile(hdfsLocation,localLocation);
>>  indiste the cleanup of mapper task , but it is failing .
>>
>> Exception file not found.
>>
>> But if I am giving same statement after the job complete in driver class
>> ,it is working fine. that i dont want.
>>
>> protected void cleanup(Context context){
>>    FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
>> }
>>
>> Note: I need to copy the inter mediate output of the mapper to local file
>> system just after  each map task complete. I dont want any reducer.
>>
>>    If this is not the right solution then how to solve this type of
>> scenario.
>>
>> Any help.............
>>
>> regards,
>> samir.
>>
>
>
+
Shahab Yunus 2013-06-03, 22:38