Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Exception in hadoop jobtracker OOM


Copy link to this message
-
Exception in hadoop jobtracker OOM
Hi,

I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
any issues.

Suddenly I got the below error in Jobtracker and jobs are failed to run.

Is this issue in JT or TT or Jetty issue?

2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
org.mortbay.io.nio. SelectorManager$SelectSet.
doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
SelectorManager.doSelect( SelectorManager.java:192) at
org.mortbay.jetty.nio. SelectChannelConnector.accept(
SelectChannelConnector.java:124) at org.mortbay.jetty.
AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
org.mortbay.thread. QueuedThreadPool$PoolThread.
run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
org.apache.hadoop.security. UserGroupInformation:
PriviledgedActionException as:root cause:java.io.IOException:
java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
heap space at java.lang.String.toCharArray( String.java:2725) at
sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
loadClass(ClassLoader.java:306) at sun.misc.Launcher$
AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
org.mortbay.jetty.servlet. ServletHandler$CachedChain.
doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
org.mortbay.jetty.servlet. ServletHandler$CachedChain.
doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
ServletHandler.handle(ServletHandler.java:399) at
org.mortbay.jetty.security. SecurityHandler.handle(
SecurityHandler.java:216) at org.mortbay.jetty.servlet.
SessionHandler.handle( SessionHandler.java:182) at
org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
HandlerWrapper.handle( HandlerWrapper.java:152) at
org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
HttpConnection.handleRequest( HttpConnection.java:542) at
org.mortbay.jetty. HttpConnection$RequestHandler.
headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
parseAvailable(HttpParser.java:212) at
org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
org.mortbay.io.nio. SelectChannelEndPoint.run(
SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
canonical subfolder. java.lang.OutOfMemoryError: Java heap space

Please help as I'm facing in production.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB