Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # user >> agent.log  I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out


Copy link to this message
-
RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out


Hi Saisai:

 

I have the hbase-site.xml in the etc/chukwa folder.

 

I upgraded to 0.6.0, the exception is gone,

 

Thanks,

Suresh

 

From: Shao, Saisai [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, September 19, 2012 5:10 PM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

 

Hi Suresh,

Removing disabled file can only fix this exception(WidgetBean - java.lang.NullPointerException).

For the zookeeper problem, I agreed with you that it is originated from HBase, but I have no clues about this exception. Did you config HBase related environment variables in chukwa-env.sh, or did you copy hbase-site.xml to etc/chukwa folder.

My chukwa version 0.6.0, currently trunk version, and my HBase version is 0.94.0. Sorry for not helping you with the zookeeper problems.

 

Thanks

Saisai

 

From: Kumar, Suresh [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 20, 2012 4:27 AM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

 

Thanks Saisai, I removed the disabled file, still see the exception.

 

What version of chukwa are you running? What version of HBase are you running?

Looking at the exception in hicc.out, looks like it is a Zookeeper exception which probably

originates from HBase.

 

2012-09-19 14:57:17,139 INFO 2124414527@qtp-1510130526-7-SendThread(localhost:2181) ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x139df9f4b070005, negotiated timeout = 40000

2012-09-19 14:57:19,142 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw

java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING

  

 

Suresh

 

 

From: Shao, Saisai [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, September 18, 2012 6:06 PM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

 

Hi Suresh,

I met this exception before.

ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException

        at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)

        at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)

        at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)

        at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)

        at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)

       at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)

        at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)

        at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)

        at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)

        at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)

I think this is a bug in HICC while reading descriptors from HDFS, at that time I delete this file (/chukwa/hicc/widgets/disabled) on hdfs and restart HICC, you can try it.

I’m sorry for the other Zookeeper exceptions, also In my environment, HICC only take several minutes to load.

 

Thanks

Saisai

 

 

From: Kumar, Suresh [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, September 19, 2012 1:45 AM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

 

I had checked these before posting, the problem was in the “initial_adaptors” file, I had copied

“add FileTailer FooData /tmp/foo 0” from the Chukwa agent documentation, it didn’t work. I had

to change to  “add filetailer.FileTailingAdaptor FooData /tmp/foo 0” to make it work. I found the

exception in “agent.log” and fixed the problem.

 

However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any clues?

 

Thanks,

Suresh

 

2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw

java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING

        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)

        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)

        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)

        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)

        at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)

        at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)

        at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)

        at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)

        at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)

        at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)

        at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)

        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)

        at javax.servlet.http.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB