Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa, mail # user - agent.log  I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out


+
Kumar, Suresh 2012-09-17, 21:00
+
邵赛赛 2012-09-18, 01:28
+
Kumar, Suresh 2012-09-18, 17:45
+
Shao, Saisai 2012-09-19, 01:06
+
Kumar, Suresh 2012-09-19, 20:26
Copy link to this message
-
RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Shao, Saisai 2012-09-20, 00:10
Hi Suresh,
Removing disabled file can only fix this exception(WidgetBean - java.lang.NullPointerException).
For the zookeeper problem, I agreed with you that it is originated from HBase, but I have no clues about this exception. Did you config HBase related environment variables in chukwa-env.sh, or did you copy hbase-site.xml to etc/chukwa folder.
My chukwa version 0.6.0, currently trunk version, and my HBase version is 0.94.0. Sorry for not helping you with the zookeeper problems.

Thanks
Saisai

From: Kumar, Suresh [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 20, 2012 4:27 AM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

Thanks Saisai, I removed the disabled file, still see the exception.

What version of chukwa are you running? What version of HBase are you running?
Looking at the exception in hicc.out, looks like it is a Zookeeper exception which probably
originates from HBase.

2012-09-19 14:57:17,139 INFO 2124414527@qtp-1510130526-7-SendThread(localhost:2181) ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x139df9f4b070005, negotiated timeout = 40000
2012-09-19 14:57:19,142 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
Suresh
From: Shao, Saisai [mailto:[EMAIL PROTECTED]]<mailto:[mailto:[EMAIL PROTECTED]]>
Sent: Tuesday, September 18, 2012 6:06 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

Hi Suresh,
I met this exception before.
ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException
        at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
        at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
        at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
        at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
        at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
       at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
        at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
        at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
        at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
        at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)
I think this is a bug in HICC while reading descriptors from HDFS, at that time I delete this file (/chukwa/hicc/widgets/disabled) on hdfs and restart HICC, you can try it.
I’m sorry for the other Zookeeper exceptions, also In my environment, HICC only take several minutes to load.

Thanks
Saisai
From: Kumar, Suresh [mailto:[EMAIL PROTECTED]]<mailto:[mailto:[EMAIL PROTECTED]]>
Sent: Wednesday, September 19, 2012 1:45 AM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

I had checked these before posting, the problem was in the “initial_adaptors” file, I had copied
“add FileTailer FooData /tmp/foo 0” from the Chukwa agent documentation, it didn’t work. I had
to change to  “add filetailer.FileTailingAdaptor FooData /tmp/foo 0” to make it work. I found the
exception in “agent.log” and fixed the problem.

However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any clues?

Thanks,
Suresh

2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)
        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)
        at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)
        at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)
        at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)
        at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)
        at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)
        at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)
        at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
        at javax.servlet.http.HttpServlet
+
Kumar, Suresh 2012-09-21, 22:17