Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Need help optimizing reducer


Copy link to this message
-
Re: Need help optimizing reducer
Are you using combiner ? If not, that will be first thing to do.
On 05-Mar-2013, at 1:27 AM, Austin Chungath wrote:

> Hi all,
>
> I have 1 reducer and I have around 600 thousand unique keys coming to it. The total data is only around 30 mb.
> My logic doesn't allow me to have more than 1 reducer.
> It's taking too long to complete, around 2 hours. (till 66% it's fast then it slows down/ I don't really think it has started doing anything till 66% but then why does it show like that?).
> Are there any job execution parameters that can help improve reducer performace?
> Any suggestions to improve things when we have to live with just one reducer?
>
> thanks,
> Austin