Hello hadoopers!

In a reduce-only Hadoop job input files are handled by the identity mapper and sent to the reducers without modification. In one of my job I was surprised to see the job failing in the map phase with "Out of memory error" and "GC overhead limit exceeded".

In my understanding, a memory leak on the identity mapper is out of the question.
What can be the cause of such error?

Thanks,
Ximo.

P.S. The logs show no stack trace other than the messages I mentioned before.

________________________________
Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra pol?tica de env?o y recepci?n de correo electr?nico en el enlace situado m?s abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx

I can imagine a huge record size possibly causing this. Is this
reliably reproducible? Do you also have combiners enabled, which may
run the reducer-logic on the map-side itself?

On Thu, Sep 6, 2012 at 8:20 PM, JOAQUIN GUANTER GONZALBEZ <[EMAIL PROTECTED]> wrote:
> Hello hadoopers!
>
>
>
> In a reduce-only Hadoop job input files are handled by the identity mapper
> and sent to the reducers without modification. In one of my job I was
> surprised to see the job failing in the map phase with "Out of memory error"
> and "GC overhead limit exceeded".
>
>
>
> In my understanding, a memory leak on the identity mapper is out of the
> question.
>
> What can be the cause of such error?
>
>
>
> Thanks,
>
> Ximo.
>
>
>
> P.S. The logs show no stack trace other than the messages I mentioned
> before.
>
>
> ________________________________
> Este mensaje se dirige exclusivamente a su destinatario. Puede consultar
> nuestra política de envío y recepción de correo electrónico en el enlace
> situado más abajo.
> This message is intended exclusively for its addressee. We only send and
> receive email on the basis of the terms set out at:
> http://www.tid.es/ES/PAGINAS/disclaimer.aspx

--
Harsh J