Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> java.lang.OutOfMemoryError: GC overhead limit exceeded


Copy link to this message
-
Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
Have you tried lowering mapred.job.reuse.jvm.num.tasks ?

On Sun, Sep 26, 2010 at 3:30 AM, Bradford Stephens <
[EMAIL PROTECTED]> wrote:

> Nope, that didn't seem to help.
>
> On Sun, Sep 26, 2010 at 1:00 AM, Bradford Stephens
> <[EMAIL PROTECTED]> wrote:
> > I'm going to try running it on high-RAM boxes with -Xmx4096m or so,
> > see if that helps.
> >
> > On Sun, Sep 26, 2010 at 12:55 AM, Bradford Stephens
> > <[EMAIL PROTECTED]> wrote:
> >> Greetings,
> >>
> >> I'm running into a brain-numbing problem on Elastic MapReduce. I'm
> >> running a decent-size task (22,000 mappers, a ton of GZipped input
> >> blocks, ~1TB of data) on 40 c1.xlarge nodes (7 gb RAM, ~8 "cores").
> >>
> >> I get failures randomly --- sometimes at the end of my 6-step process,
> >> sometimes at the first reducer phase, sometimes in the mapper. It
> >> seems to fail in multiple areas. Mostly in the reducers. Any ideas?
> >>
> >> Here's the settings I've changed:
> >> -Xmx400m
> >> 6 max mappers
> >> 1 max reducer
> >> 1GB swap partition
> >> mapred.job.reuse.jvm.num.tasks=50
> >> mapred.reduce.parallel.copies=3
> >>
> >>
> >> java.lang.OutOfMemoryError: GC overhead limit exceeded
> >>        at java.nio.CharBuffer.wrap(CharBuffer.java:350)
> >>        at java.nio.CharBuffer.wrap(CharBuffer.java:373)
> >>        at
> java.lang.StringCoding$StringDecoder.decode(StringCoding.java:138)
> >>        at java.lang.StringCoding.decode(StringCoding.java:173)
> >>        at java.lang.String.(String.java:443)
> >>        at java.lang.String.(String.java:515)
> >>        at
> org.apache.hadoop.io.WritableUtils.readString(WritableUtils.java:116)
> >>        at
> cascading.tuple.TupleInputStream.readString(TupleInputStream.java:144)
> >>        at
> cascading.tuple.TupleInputStream.readType(TupleInputStream.java:154)
> >>        at
> cascading.tuple.TupleInputStream.getNextElement(TupleInputStream.java:101)
> >>        at
> cascading.tuple.hadoop.TupleElementComparator.compare(TupleElementComparator.java:75)
> >>        at
> cascading.tuple.hadoop.TupleElementComparator.compare(TupleElementComparator.java:33)
> >>        at
> cascading.tuple.hadoop.DelegatingTupleElementComparator.compare(DelegatingTupleElementComparator.java:74)
> >>        at
> cascading.tuple.hadoop.DelegatingTupleElementComparator.compare(DelegatingTupleElementComparator.java:34)
> >>        at
> cascading.tuple.hadoop.DeserializerComparator.compareTuples(DeserializerComparator.java:142)
> >>        at
> cascading.tuple.hadoop.GroupingSortingComparator.compare(GroupingSortingComparator.java:55)
> >>        at
> org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:373)
> >>        at
> org.apache.hadoop.util.PriorityQueue.downHeap(PriorityQueue.java:136)
> >>        at
> org.apache.hadoop.util.PriorityQueue.adjustTop(PriorityQueue.java:103)
> >>        at
> org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:335)
> >>        at
> org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350)
> >>        at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:156)
> >>        at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2645)
> >>        at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2586)
> >>
> >> --
> >> Bradford Stephens,
> >> Founder, Drawn to Scale
> >> drawntoscalehq.com
> >> 727.697.7528
> >>
> >> http://www.drawntoscalehq.com --  The intuitive, cloud-scale data
> >> solution. Process, store, query, search, and serve all your data.
> >>
> >> http://www.roadtofailure.com -- The Fringes of Scalability, Social
> >> Media, and Computer Science
> >>
> >
> >
> >
> > --
> > Bradford Stephens,
> > Founder, Drawn to Scale
> > drawntoscalehq.com
> > 727.697.7528
> >
> > http://www.drawntoscalehq.com --  The intuitive, cloud-scale data
> > solution. Process, store, query, search, and serve all your data.
> >
> > http://www.roadtofailure.com -- The Fringes of Scalability, Social
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB