Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Exception in hadoop jobtracker OOM


Copy link to this message
-
Exception in hadoop jobtracker OOM
Viswanathan J 2014-04-12, 15:03
Hi,

I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without
any issues.

Suddenly I got the below error in Jobtracker and jobs are failed to run.

Is this issue in JT or TT or Jetty issue?

2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION
java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.
newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.
iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153)
at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127)
at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at
sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at
sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at
org.mortbay.io.nio. SelectorManager$SelectSet.
doSelect(SelectorManager.java:498) at org.mortbay.io.nio.
SelectorManager.doSelect( SelectorManager.java:192) at
org.mortbay.jetty.nio. SelectChannelConnector.accept(
SelectChannelConnector.java:124) at org.mortbay.jetty.
AbstractConnector$Acceptor. run(AbstractConnector.java:708) at
org.mortbay.thread. QueuedThreadPool$PoolThread.
run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR
org.apache.hadoop.security. UserGroupInformation:
PriviledgedActionException as:root cause:java.io.IOException:
java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR
org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java
heap space at java.lang.String.toCharArray( String.java:2725) at
sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at
sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at
sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at
sun.misc.URLClassPath. getResource(URLClassPath.java:169) at
java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security.
AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.
findClass(URLClassLoader.java:190) at java.lang.ClassLoader.
loadClass(ClassLoader.java:306) at sun.misc.Launcher$
AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.
loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.
jobtracker_jsp._jspService( jobtracker_jsp.java:508) at
org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at
javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at
org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at
org.mortbay.jetty.servlet. ServletHandler$CachedChain.
doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.
HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at
org.mortbay.jetty.servlet. ServletHandler$CachedChain.
doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.
ServletHandler.handle(ServletHandler.java:399) at
org.mortbay.jetty.security. SecurityHandler.handle(
SecurityHandler.java:216) at org.mortbay.jetty.servlet.
SessionHandler.handle( SessionHandler.java:182) at
org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766)
at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450)
at org.mortbay.jetty.handler. ContextHandlerCollection.handle(
ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.
HandlerWrapper.handle( HandlerWrapper.java:152) at
org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty.
HttpConnection.handleRequest( HttpConnection.java:542) at
org.mortbay.jetty. HttpConnection$RequestHandler.
headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.
parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.
parseAvailable(HttpParser.java:212) at
org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at
org.mortbay.io.nio. SelectChannelEndPoint.run(
SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR
org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
canonical subfolder. java.lang.OutOfMemoryError: Java heap space

Please help as I'm facing in production.