Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Reducer not firing


Copy link to this message
-
RE: Reducer not firing
Can you check the task attempt logs in your cluster and find out what is happening in the reduce phase. By default task attempt logs present in $HADOOP_LOG_DIR/userlogs/<job-id>/. There could be some bug exist in your reducer which is leading to this output.

Thanks
Devaraj

________________________________________
From: Arko Provo Mukherjee [[EMAIL PROTECTED]]
Sent: Tuesday, April 17, 2012 2:07 PM
To: [EMAIL PROTECTED]
Subject: Re: Reducer not firing

Hello,

Many thanks for the reply.

The 'no_of_reduce_tasks' is set to 2. I have a print statement before
the code I pasted below to check that.

Also I can find two output files part-r-00000 and part-r-00001. But
they contain the values that has been outputted by the Mapper logic.

Please let me know what I can check further.

Thanks a lot in advance!

Warm regards
Arko

On Tue, Apr 17, 2012 at 12:48 AM, Devaraj k <[EMAIL PROTECTED]> wrote:
> Hi Arko,
>
>    What is value of  'no_of_reduce_tasks'?
>
> If no of reduce tasks are 0, then the map task will directly write map output  into the Job output path.
>
> Thanks
> Devaraj
>
> ________________________________________
> From: Arko Provo Mukherjee [[EMAIL PROTECTED]]
> Sent: Tuesday, April 17, 2012 10:32 AM
> To: [EMAIL PROTECTED]
> Subject: Reducer not firing
>
> Dear All,
>
> I am porting code from the old API to the new API (Context objects)
> and run on Hadoop 0.20.203.
>
> Job job_first = new Job();
>
> job_first.setJarByClass(My.class);
> job_first.setNumReduceTasks(no_of_reduce_tasks);
> job_first.setJobName("My_Job");
>
> FileInputFormat.addInputPath( job_first, new Path (Input_Path) );
> FileOutputFormat.setOutputPath( job_first, new Path (Output_Path) );
>
> job_first.setMapperClass(Map_First.class);
> job_first.setReducerClass(Reduce_First.class);
>
> job_first.setMapOutputKeyClass(IntWritable.class);
> job_first.setMapOutputValueClass(Text.class);
>
> job_first.setOutputKeyClass(NullWritable.class);
> job_first.setOutputValueClass(Text.class);
>
> job_first.waitForCompletion(true);
>
> The problem I am facing is that instead of emitting values to
> reducers, the mappers are directly writing their output in the
> OutputPath and the reducers and not processing anything.
>
> As read from the online materials that are available both my Map and
> Reduce method uses the context.write method to emit the values.
>
> Please help. Thanks a lot in advance!!
>
> Warm regards
> Arko
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB