Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Reducer not firing

Copy link to this message
RE: Reducer not firing
Can you check the task attempt logs in your cluster and find out what is happening in the reduce phase. By default task attempt logs present in $HADOOP_LOG_DIR/userlogs/<job-id>/. There could be some bug exist in your reducer which is leading to this output.


From: Arko Provo Mukherjee [[EMAIL PROTECTED]]
Sent: Tuesday, April 17, 2012 2:07 PM
Subject: Re: Reducer not firing


Many thanks for the reply.

The 'no_of_reduce_tasks' is set to 2. I have a print statement before
the code I pasted below to check that.

Also I can find two output files part-r-00000 and part-r-00001. But
they contain the values that has been outputted by the Mapper logic.

Please let me know what I can check further.

Thanks a lot in advance!

Warm regards

On Tue, Apr 17, 2012 at 12:48 AM, Devaraj k <[EMAIL PROTECTED]> wrote:
> Hi Arko,
>    What is value of  'no_of_reduce_tasks'?
> If no of reduce tasks are 0, then the map task will directly write map output  into the Job output path.
> Thanks
> Devaraj
> ________________________________________
> From: Arko Provo Mukherjee [[EMAIL PROTECTED]]
> Sent: Tuesday, April 17, 2012 10:32 AM
> Subject: Reducer not firing
> Dear All,
> I am porting code from the old API to the new API (Context objects)
> and run on Hadoop 0.20.203.
> Job job_first = new Job();
> job_first.setJarByClass(My.class);
> job_first.setNumReduceTasks(no_of_reduce_tasks);
> job_first.setJobName("My_Job");
> FileInputFormat.addInputPath( job_first, new Path (Input_Path) );
> FileOutputFormat.setOutputPath( job_first, new Path (Output_Path) );
> job_first.setMapperClass(Map_First.class);
> job_first.setReducerClass(Reduce_First.class);
> job_first.setMapOutputKeyClass(IntWritable.class);
> job_first.setMapOutputValueClass(Text.class);
> job_first.setOutputKeyClass(NullWritable.class);
> job_first.setOutputValueClass(Text.class);
> job_first.waitForCompletion(true);
> The problem I am facing is that instead of emitting values to
> reducers, the mappers are directly writing their output in the
> OutputPath and the reducers and not processing anything.
> As read from the online materials that are available both my Map and
> Reduce method uses the context.write method to emit the values.
> Please help. Thanks a lot in advance!!
> Warm regards
> Arko