Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Reducer not firing


+
Arko Provo Mukherjee 2012-04-17, 05:02
+
Devaraj k 2012-04-17, 05:48
+
Arko Provo Mukherjee 2012-04-17, 08:37
+
Devaraj k 2012-04-17, 09:30
+
kasi subrahmanyam 2012-04-17, 13:40
+
Bejoy KS 2012-04-17, 14:03
+
Steven Willis 2012-04-17, 20:19
+
Arko Provo Mukherjee 2012-04-17, 23:16
Copy link to this message
-
Re: Reducer not firing
Arko,

Change Iterator to Iterable
George
On 2012/04/18 8:16, Arko Provo Mukherjee wrote:
> Hello,
>
> Thanks everyone for helping me. Here are my observations:
>
> Devaraj - I didn't find any bug in the log files. In fact, none of the
> print statements in my reducer are even appearing in the logs. I can
> share the syslogs if you want. I didn't paste them here so that the
> email doesn't get cluttered.
>
> Kasi -  Thanks for the suggestion. I tired but got the same output.
> The system just created 1 reducer as my test data set is small.
>
> Bejoy -  Can you please advice how I can pinpoint whether the
> IdentityReducer is being used or not.
>
> Steven - I tried compiling with your suggestion. However if I put a
> @Override on top of my reduce method, I get the following error:
> "method does not override or implement a method from a supertype"
> The code compiles without it. I do have an @Override on top of my map
> method though.
> public class Reduce_First extends Reducer<IntWritable, Text,
> NullWritable, Text>
> {
>      public void reduce (IntWritable key, Iterator<Text>  values,
> Context context) throws IOException, InterruptedException
>      {
>          while ( values.hasNext() )
>                 // Process
>
>          // Finally emit
>      }
> }
>
> Thanks a lot again!
> Warm regards
> Arko
>
>
> On Tue, Apr 17, 2012 at 3:19 PM, Steven Willis<[EMAIL PROTECTED]>  wrote:
>> Try putting @Override before your reduce method to make sure you're
>> overriding the method properly. You�ll get a compile time error if not.
>>
>>
>>
>> -Steven Willis
>>
>>
>>
>>
>>
>> From: Bejoy KS [mailto:[EMAIL PROTECTED]]
>> Sent: Tuesday, April 17, 2012 10:03 AM
>>
>>
>> To: [EMAIL PROTECTED]
>> Subject: Re: Reducer not firing
>>
>>
>>
>> Hi Akro
>>  From the naming of output files, your job has the reduce phase. But the
>> reducer being used is the IdentityReducer instead of your custom reducer.
>> That is the reason you are seeing the same map output in the output files as
>> well. You need to evaluate your code and logs to see why IdentityReducer is
>> being triggered.
>>
>> Regards
>> Bejoy KS
>>
>> Sent from handheld, please excuse typos.
>>
>> ________________________________
>>
>> From: kasi subrahmanyam<[EMAIL PROTECTED]>
>>
>> Date: Tue, 17 Apr 2012 19:10:33 +0530
>>
>> To:<[EMAIL PROTECTED]>
>>
>> ReplyTo: [EMAIL PROTECTED]
>>
>> Subject: Re: Reducer not firing
>>
>>
>>
>> Could you comment the property where you are setting the number of reducer
>> tasks and see the behaviour of the program once.
>> If you already tried could you share the output
>>
>> On Tue, Apr 17, 2012 at 3:00 PM, Devaraj k<[EMAIL PROTECTED]>  wrote:
>>
>> Can you check the task attempt logs in your cluster and find out what is
>> happening in the reduce phase. By default task attempt logs present in
>> $HADOOP_LOG_DIR/userlogs/<job-id>/. There could be some bug exist in your
>> reducer which is leading to this output.
>>
>>
>> Thanks
>> Devaraj
>>
>> ________________________________________
>> From: Arko Provo Mukherjee [[EMAIL PROTECTED]]
>>
>> Sent: Tuesday, April 17, 2012 2:07 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Reducer not firing
>>
>>
>> Hello,
>>
>> Many thanks for the reply.
>>
>> The 'no_of_reduce_tasks' is set to 2. I have a print statement before
>> the code I pasted below to check that.
>>
>> Also I can find two output files part-r-00000 and part-r-00001. But
>> they contain the values that has been outputted by the Mapper logic.
>>
>> Please let me know what I can check further.
>>
>> Thanks a lot in advance!
>>
>> Warm regards
>> Arko
>>
>> On Tue, Apr 17, 2012 at 12:48 AM, Devaraj k<[EMAIL PROTECTED]>  wrote:
>>> Hi Arko,
>>>
>>>     What is value of  'no_of_reduce_tasks'?
>>>
>>> If no of reduce tasks are 0, then the map task will directly write map
>>> output  into the Job output path.
>>>
>>> Thanks
>>> Devaraj
>>>
>>> ________________________________________
+
Arko Provo Mukherjee 2012-04-18, 00:22
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB