Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Reducer not firing


Copy link to this message
-
RE: Reducer not firing
Devaraj k 2012-04-17, 09:30
Can you check the task attempt logs in your cluster and find out what is happening in the reduce phase. By default task attempt logs present in $HADOOP_LOG_DIR/userlogs/<job-id>/. There could be some bug exist in your reducer which is leading to this output.

Thanks
Devaraj

________________________________________
From: Arko Provo Mukherjee [[EMAIL PROTECTED]]
Sent: Tuesday, April 17, 2012 2:07 PM
To: [EMAIL PROTECTED]
Subject: Re: Reducer not firing

Hello,

Many thanks for the reply.

The 'no_of_reduce_tasks' is set to 2. I have a print statement before
the code I pasted below to check that.

Also I can find two output files part-r-00000 and part-r-00001. But
they contain the values that has been outputted by the Mapper logic.

Please let me know what I can check further.

Thanks a lot in advance!

Warm regards
Arko

On Tue, Apr 17, 2012 at 12:48 AM, Devaraj k <[EMAIL PROTECTED]> wrote:
> Hi Arko,
>
>    What is value of  'no_of_reduce_tasks'?
>
> If no of reduce tasks are 0, then the map task will directly write map output  into the Job output path.
>
> Thanks
> Devaraj
>
> ________________________________________
> From: Arko Provo Mukherjee [[EMAIL PROTECTED]]
> Sent: Tuesday, April 17, 2012 10:32 AM
> To: [EMAIL PROTECTED]
> Subject: Reducer not firing
>
> Dear All,
>
> I am porting code from the old API to the new API (Context objects)
> and run on Hadoop 0.20.203.
>
> Job job_first = new Job();
>
> job_first.setJarByClass(My.class);
> job_first.setNumReduceTasks(no_of_reduce_tasks);
> job_first.setJobName("My_Job");
>
> FileInputFormat.addInputPath( job_first, new Path (Input_Path) );
> FileOutputFormat.setOutputPath( job_first, new Path (Output_Path) );
>
> job_first.setMapperClass(Map_First.class);
> job_first.setReducerClass(Reduce_First.class);
>
> job_first.setMapOutputKeyClass(IntWritable.class);
> job_first.setMapOutputValueClass(Text.class);
>
> job_first.setOutputKeyClass(NullWritable.class);
> job_first.setOutputValueClass(Text.class);
>
> job_first.waitForCompletion(true);
>
> The problem I am facing is that instead of emitting values to
> reducers, the mappers are directly writing their output in the
> OutputPath and the reducers and not processing anything.
>
> As read from the online materials that are available both my Map and
> Reduce method uses the context.write method to emit the values.
>
> Please help. Thanks a lot in advance!!
>
> Warm regards
> Arko