Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - Out of memory in identity mapper?


+
JOAQUIN GUANTER GONZALBEZ... 2012-09-06, 14:50
Copy link to this message
-
Re: Out of memory in identity mapper?
Harsh J 2012-09-06, 15:13
I can imagine a huge record size possibly causing this. Is this
reliably reproducible? Do you also have combiners enabled, which may
run the reducer-logic on the map-side itself?

On Thu, Sep 6, 2012 at 8:20 PM, JOAQUIN GUANTER GONZALBEZ <[EMAIL PROTECTED]> wrote:
> Hello hadoopers!
>
>
>
> In a reduce-only Hadoop job input files are handled by the identity mapper
> and sent to the reducers without modification. In one of my job I was
> surprised to see the job failing in the map phase with "Out of memory error"
> and "GC overhead limit exceeded".
>
>
>
> In my understanding, a memory leak on the identity mapper is out of the
> question.
>
> What can be the cause of such error?
>
>
>
> Thanks,
>
> Ximo.
>
>
>
> P.S. The logs show no stack trace other than the messages I mentioned
> before.
>
>
> ________________________________
> Este mensaje se dirige exclusivamente a su destinatario. Puede consultar
> nuestra política de envío y recepción de correo electrónico en el enlace
> situado más abajo.
> This message is intended exclusively for its addressee. We only send and
> receive email on the basis of the terms set out at:
> http://www.tid.es/ES/PAGINAS/disclaimer.aspx

--
Harsh J