Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Reducer not firing


+
Arko Provo Mukherjee 2012-04-17, 05:02
+
Devaraj k 2012-04-17, 05:48
+
Arko Provo Mukherjee 2012-04-17, 08:37
+
Devaraj k 2012-04-17, 09:30
+
kasi subrahmanyam 2012-04-17, 13:40
+
Bejoy KS 2012-04-17, 14:03
+
Steven Willis 2012-04-17, 20:19
+
Arko Provo Mukherjee 2012-04-17, 23:16
Copy link to this message
-
Re: Reducer not firing
Arko,

Change Iterator to Iterable
George
On 2012/04/18 8:16, Arko Provo Mukherjee wrote:
> Hello,
>
> Thanks everyone for helping me. Here are my observations:
>
> Devaraj - I didn't find any bug in the log files. In fact, none of the
> print statements in my reducer are even appearing in the logs. I can
> share the syslogs if you want. I didn't paste them here so that the
> email doesn't get cluttered.
>
> Kasi -  Thanks for the suggestion. I tired but got the same output.
> The system just created 1 reducer as my test data set is small.
>
> Bejoy -  Can you please advice how I can pinpoint whether the
> IdentityReducer is being used or not.
>
> Steven - I tried compiling with your suggestion. However if I put a
> @Override on top of my reduce method, I get the following error:
> "method does not override or implement a method from a supertype"
> The code compiles without it. I do have an @Override on top of my map
> method though.
> public class Reduce_First extends Reducer<IntWritable, Text,
> NullWritable, Text>
> {
>      public void reduce (IntWritable key, Iterator<Text>  values,
> Context context) throws IOException, InterruptedException
>      {
>          while ( values.hasNext() )
>                 // Process
>
>          // Finally emit
>      }
> }
>
> Thanks a lot again!
> Warm regards
> Arko
>
>
> On Tue, Apr 17, 2012 at 3:19 PM, Steven Willis<[EMAIL PROTECTED]>  wrote:
>> Try putting @Override before your reduce method to make sure you're
>> overriding the method properly. You�ll get a compile time error if not.
>>
>>
>>
>> -Steven Willis
>>
>>
>>
>>
>>
>> From: Bejoy KS [mailto:[EMAIL PROTECTED]]
>> Sent: Tuesday, April 17, 2012 10:03 AM
>>
>>
>> To: [EMAIL PROTECTED]
>> Subject: Re: Reducer not firing
>>
>>
>>
>> Hi Akro
>>  From the naming of output files, your job has the reduce phase. But the
>> reducer being used is the IdentityReducer instead of your custom reducer.
>> That is the reason you are seeing the same map output in the output files as
>> well. You need to evaluate your code and logs to see why IdentityReducer is
>> being triggered.
>>
>> Regards
>> Bejoy KS
>>
>> Sent from handheld, please excuse typos.
>>
>> ________________________________
>>
>> From: kasi subrahmanyam<[EMAIL PROTECTED]>
>>
>> Date: Tue, 17 Apr 2012 19:10:33 +0530
>>
>> To:<[EMAIL PROTECTED]>
>>
>> ReplyTo: [EMAIL PROTECTED]
>>
>> Subject: Re: Reducer not firing
>>
>>
>>
>> Could you comment the property where you are setting the number of reducer
>> tasks and see the behaviour of the program once.
>> If you already tried could you share the output
>>
>> On Tue, Apr 17, 2012 at 3:00 PM, Devaraj k<[EMAIL PROTECTED]>  wrote:
>>
>> Can you check the task attempt logs in your cluster and find out what is
>> happening in the reduce phase. By default task attempt logs present in
>> $HADOOP_LOG_DIR/userlogs/<job-id>/. There could be some bug exist in your
>> reducer which is leading to this output.
>>
>>
>> Thanks
>> Devaraj
>>
>> ________________________________________
>> From: Arko Provo Mukherjee [[EMAIL PROTECTED]]
>>
>> Sent: Tuesday, April 17, 2012 2:07 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Reducer not firing
>>
>>
>> Hello,
>>
>> Many thanks for the reply.
>>
>> The 'no_of_reduce_tasks' is set to 2. I have a print statement before
>> the code I pasted below to check that.
>>
>> Also I can find two output files part-r-00000 and part-r-00001. But
>> they contain the values that has been outputted by the Mapper logic.
>>
>> Please let me know what I can check further.
>>
>> Thanks a lot in advance!
>>
>> Warm regards
>> Arko
>>
>> On Tue, Apr 17, 2012 at 12:48 AM, Devaraj k<[EMAIL PROTECTED]>  wrote:
>>> Hi Arko,
>>>
>>>     What is value of  'no_of_reduce_tasks'?
>>>
>>> If no of reduce tasks are 0, then the map task will directly write map
>>> output  into the Job output path.
>>>
>>> Thanks
>>> Devaraj
>>>
>>> ________________________________________
+
Arko Provo Mukherjee 2012-04-18, 00:22