Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> jobtracker.jsp reports "GC overhead limit exceeded"


Copy link to this message
-
Re: jobtracker.jsp reports "GC overhead limit exceeded"
Hi,

> Actually I enabled all level logs. But I didn't realize to check logs in .out files and only looked at .log file and didn't see any error msgs. now I opened the .out file and saw the following logged exception:
>
> Exception in thread "IPC Server handler 5 on 50002" java.lang.OutOfMemoryError: Java heap space
>     at java.util.Arrays.copyOf(Arrays.java:2786)
>     at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:133)
>     at org.apache.hadoop.ipc.Server.setupResponse(Server.java:1087)
>     at org.apache.hadoop.ipc.Server.access$2400(Server.java:77)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:984)
>
> Exception in thread "IPC Client (47) connection to servername/serverip:50001 from auser" java.lang.OutOfMemoryError: Java heap space
>     at java.nio.ByteBuffer.wrap(ByteBuffer.java:350)
>     at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>     at java.io.FilterInputStream.read(FilterInputStream.java:116)
>     at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276)
>     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>     at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>     at java.io.DataInputStream.readInt(DataInputStream.java:370)
>     at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
>     at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
>
> Exception in thread "LeaseChecker" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "IPC Server handler 4 on 50002" java.lang.OutOfMemoryError: Java heap space
> Exception in thread "IPC Server handler 8 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "IPC Server handler 9 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "IPC Server handler 2 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "IPC Server handler 6 on 50002" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "ResponseProcessor for block blk_-573819335330670501_1268378" java.lang.OutOfMemoryError: GC overhead limit exceeded
> Exception in thread "IPC Server handler 7 on 50002" java.lang.OutOfMemoryError: Java heap space
>
> 50001 and 50002 used for dfs and jobtracker. But it's normal that IPC threads consume such big memory?
>

It may not be IPC threads that are causing the problem, though I
remember at least one IPC buffer related OOM somewhere. A more typical
case is that the daemon is running out of memory and the IPC threads
are just manifesting the error because they're the entry-points into
the application. I think we still need to go down the path that Alex
was suggesting. What is the heap you have given the JobTracker ? What
is number of Jobs / tasks / users ? How many nodes are there in the
cluster ?

>
> --- On Fri, 7/30/10, Alex Loddengaard <[EMAIL PROTECTED]> wrote:
>
> From: Alex Loddengaard <[EMAIL PROTECTED]>
> Subject: Re: jobtracker.jsp reports "GC overhead limit exceeded"
> To: [EMAIL PROTECTED]
> Date: Friday, July 30, 2010, 2:19 PM
>
> What does "ps" show you?  How much memory is being used by the jobtracker,
> and how large is its heap (loop for HADOOP_HEAPSIZE in hadoop-env.sh)?  Also
> consider turning on GC logging, which will find its way to the jobtracker
> .out log in /var/log/hadoop:
>
> <http://java.sun.com/developer/technicalArticles/Programming/GCPortal/>
>
> Alex
>
> On Fri, Jul 30, 2010 at 3:10 PM, jiang licht <[EMAIL PROTECTED]> wrote:
>
>> http://server:50030/jobtracker.jsp generates the following error message:
>>
>> HTTP ERROR: 500
>>
>> GC overhead limit exceeded
>>
>> RequestURI=/jobtracker.jsp
>> Caused by:
>>
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>
>> Powered by Jetty://
>>
>> The jobtracker is running below the limit. But "hadoop job -status" seems
>> to halt and does not response ...
>>
>> The last 2 lines of jobtracker logs:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB