Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Killed : GC overhead limit exceeded


Copy link to this message
-
Re: Killed : GC overhead limit exceeded
Ted Yu 2010-07-17, 05:28
Have you tried increasing memory beyond 1GB for your map task ?

I think you have noticed that both OOME came from Pattern.compile().

Please take a look at
http://www.docjar.com/html/api/java/lang/String.java.html

I would suggest pre-compiling the three patterns when setting up your mapper
- basically write your own split() and replaceAll().

I recently did something similar. You can find out the performance
improvement by customization -
https://issues.apache.org/jira/browse/MAPREDUCE-1946

Cheers

On Fri, Jul 16, 2010 at 6:06 AM, Some Body <[EMAIL PROTECTED]> wrote:

> Guess attachments are stripped.
>
> Here's the memory graph:   http://tinyurl.com/37g3hmu
> Here's the VM Summary:   http://tinyurl.com/36wqzjq
>
> Alan
>