Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # user >> agent.log  I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out


Copy link to this message
-
RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Thanks Saisai, I removed the disabled file, still see the exception.

 

What version of chukwa are you running? What version of HBase are you running?

Looking at the exception in hicc.out, looks like it is a Zookeeper exception which probably

originates from HBase.

 

2012-09-19 14:57:17,139 INFO 2124414527@qtp-1510130526-7-SendThread(localhost:2181) ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x139df9f4b070005, negotiated timeout = 40000

2012-09-19 14:57:19,142 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw

java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING

  

 

Suresh

 

 

From: Shao, Saisai [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, September 18, 2012 6:06 PM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

 

Hi Suresh,

I met this exception before.

ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException

        at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)

        at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)

        at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)

        at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)

        at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)

       at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)

        at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)

        at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)

        at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)

        at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)

I think this is a bug in HICC while reading descriptors from HDFS, at that time I delete this file (/chukwa/hicc/widgets/disabled) on hdfs and restart HICC, you can try it.

I’m sorry for the other Zookeeper exceptions, also In my environment, HICC only take several minutes to load.

 

Thanks

Saisai

 

 

From: Kumar, Suresh [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, September 19, 2012 1:45 AM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

 

I had checked these before posting, the problem was in the “initial_adaptors” file, I had copied

“add FileTailer FooData /tmp/foo 0” from the Chukwa agent documentation, it didn’t work. I had

to change to  “add filetailer.FileTailingAdaptor FooData /tmp/foo 0” to make it work. I found the

exception in “agent.log” and fixed the problem.

 

However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any clues?

 

Thanks,

Suresh

 

2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw

java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING

        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)

        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)

        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)

        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)

        at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)

        at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)

        at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)

        at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)

        at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)

        at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)

        at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)

        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)

        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)

        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)

        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)

        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)

        at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)

        at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)

        at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)

        at javax.servlet.http.HttpServlet.serv