Chukwa, mail # user - agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

Kumar, Suresh 2012-09-17, 21:00

I have the following in my local host :

hadoop-1.0.3

chukwa-incubating-0.5.0

hbase-0.92.2

Here are some relevant properties from chukwa-collector-conf.xml

<property>

   <name>writer.hdfs.filesystem</name>

   <value>hdfs://localhost:9000</value>

   <description>HDFS to dump to</description>

  </property>

.........

<property>

   <name>chukwaCollector.http.port</name>

   <value>8081</value>

   <description>The HTTP port number the collector will listen
on</description>

  </property>

and collectors

http://localhost:8081

I see the following in agent.log and http://localhost:4080/hicc is
spinning, any help appreciated.

2012-09-17 16:00:28,698 INFO main Log4JMetricsContext -
Log4JMetricsContext.chukwaAgent.period=60

2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_HOME:
[/opt/chukwa-incubating-0.5.0]

2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config -
CHUKWA_CONF_DIR: [/opt/chukwa/etc/chukwa]

2012-09-17 16:00:29,619 INFO main ChukwaAgent - Config -
CHECKPOINT_BASE_NAME: [chukwa_agent_checkpoint]

2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - checkpointDir:
[/opt/chukwa/log]

2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config -
CHECKPOINT_INTERVAL_MS: [5000]

2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config -
DO_CHECKPOINT_RESTORE: [true]

2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config -
STATS_INTERVAL_MS: [10000]

2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - tags:
[cluster="demo"]

2012-09-17 16:00:29,620 INFO main ChukwaAgent - checkpoints are enabled,
period is 5000

2012-09-17 16:00:29,620 INFO main ChukwaAgent - starting adaptors listed
in /opt/chukwa/log/chukwa_agent_checkpoint0

2012-09-17 16:00:29,648 INFO main ChukwaAgent - started a new adaptor,
id = adaptor_93df4746476c9a4b624f6755b122f9dc
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@
30384065]

2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor,
id = adaptor_79203ebcc495e11b747254e660380620
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@
212bcd4b]

2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor,
id = adaptor_e2b39549cac6e966126df286a79ae6ad
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@
296f25a7]

2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor,
id = adaptor_c79bf882974a14286cffe29d3d4cf0d6
function=[org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMe
trics@a13f991]

2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor,
id = adaptor_c6fd1e5021ad8023477ef0b56a24a29f
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@
7dc05ffd]

2012-09-17 16:00:29,667 INFO main ChukwaAgent - starting adaptors listed
in /opt/chukwa/etc/chukwa/initial_adaptors

2012-09-17 16:00:29,675 INFO main AgentControlSocketListener -
AgentControlSocketListerner ask for port: 9093

2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - socket
bound to 9093

2012-09-17 16:00:29,675 INFO main ChukwaAgent - control socket started
on port 9093

2012-09-17 16:00:29,713 INFO main log - Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog

2012-09-17 16:00:29,759 INFO main log - jetty-6.1.26

2012-09-17 16:00:29,821 INFO main log - Started
SelectChannelConnector@0.0.0.0:9090

2012-09-17 16:00:29,822 INFO main ChukwaAgent - started Chukwa http
agent interface on port 9090

2012-09-17 16:00:29,826 INFO Timer-2 ChukwaAgent - writing checkpoint 1

2012-09-17 16:00:29,827 INFO main ChukwaAgent - local agent started on
port 9093

2012-09-17 16:00:29,827 INFO HTTP post thread HttpConnector -
HttpConnector started at time:1347912029827

2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config -
System.getenv("CHUKWA_HOME"): [/opt/chukwa-incubating-0.5.0/]

2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config -
System.getenv("chukwaConf"): [/opt/chukwa/etc/chukwa]

2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - setting up
collectors file: /opt/chukwa/etc/chukwa/collectors

2012-09-17 16:00:29,866 INFO HTTP post thread HttpConnector - using
collectors from collectors file

2012-09-17 16:00:29,926 INFO Timer-3 HttpConnector - # http chunks
ACK'ed since last report: 0

2012-09-17 16:00:43,886 INFO HTTP post thread ChukwaHttpSender -
collected 1 chunks for post_0

2012-09-17 16:00:43,943 INFO HTTP post thread ChukwaHttpSender - >>>>>>
HTTP post_0 to http://localhost:8081/ length = 248

2012-09-17 16:00:44,037 ERROR HTTP post thread ChukwaHttpSender - Http
post exception on http://localhost:8081/: java.net.ConnectException:
Connection refused

2012-09-17 16:00:44,037 WARN HTTP post thread ChukwaHttpSender - No more
collectors to try rolling over to; waiting 20000 ms (144000 retries
left)

2012-09-17 16:01:28,711 INFO Thread-20 SocketAdaptor - Caught
java.io.EOFException closing conneciton.

Re: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

邵赛赛 2012-09-18, 01:28

Hi, it seems like a network problem. please:
1. check whether your Chukwa Collector has been started correctly, and bind
correct port.
2. maybe port 8081 has been occupied by other process, so please check the
collecor.log to see if there is any exception.
3. check your Chukwa Agent and Collector configure to make sure it is
correct.

Thanks
Saisai
2012/9/18 Kumar, Suresh <[EMAIL PROTECTED]>

> I have the following in my local host :****
>
> ** **
>
> hadoop-1.0.3****
>
> chukwa-incubating-0.5.0****
>
> hbase-0.92.2****
>
> ** **
>
> Here are some relevant properties from *chukwa-collector-conf.xml*****
>
> ** **
>
> <property>****
>
> <name>writer.hdfs.filesystem</name>****
>
> <value>hdfs://localhost:9000</value>****
>
> <description>HDFS to dump to</description>****
>
> </property>****
>
> ………****
>
> <property>****
>
> <name>chukwaCollector.http.port</name>****
>
> <value>8081</value>****
>
> <description>The HTTP port number the collector will listen
> on</description>****
>
> </property>****
>
> ** **
>
> and *collectors*
>
> * *
>
> http://localhost:8081****
>
> ** **
>
> I see the following in agent.log and http://localhost:4080/hicc is
> spinning, any help appreciated.****
>
> ** **
>
> 2012-09-17 16:00:28,698 INFO main Log4JMetricsContext -
> Log4JMetricsContext.chukwaAgent.period=60****
>
> 2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_HOME:
> [/opt/chukwa-incubating-0.5.0]****
>
> 2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_CONF_DIR:
> [/opt/chukwa/etc/chukwa]****
>
> 2012-09-17 16:00:29,619 INFO main ChukwaAgent - Config -
> CHECKPOINT_BASE_NAME: [chukwa_agent_checkpoint]****
>
> 2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - checkpointDir:
> [/opt/chukwa/log]****
>
> 2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config -
> CHECKPOINT_INTERVAL_MS: [5000]****
>
> 2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config -
> DO_CHECKPOINT_RESTORE: [true]****
>
> 2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config -
> STATS_INTERVAL_MS: [10000]****
>
> 2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - tags:
> [cluster="demo"]****
>
> 2012-09-17 16:00:29,620 INFO main ChukwaAgent - checkpoints are enabled,
> period is 5000****
>
> 2012-09-17 16:00:29,620 INFO main ChukwaAgent - starting adaptors listed
> in /opt/chukwa/log/chukwa_agent_checkpoint0****
>
> 2012-09-17 16:00:29,648 INFO main ChukwaAgent - started a new adaptor, id
> = adaptor_93df4746476c9a4b624f6755b122f9dc
> function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@30384065
> ]****
>
> 2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id
> = adaptor_79203ebcc495e11b747254e660380620
> function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@212bcd4b
> ]****
>
> 2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id
> = adaptor_e2b39549cac6e966126df286a79ae6ad
> function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@296f25a7
> ]****
>
> 2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id
> = adaptor_c79bf882974a14286cffe29d3d4cf0d6
> function=[org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMetrics@a13f991
> ]****
>
> 2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id
> = adaptor_c6fd1e5021ad8023477ef0b56a24a29f
> function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@7dc05ffd
> ]****
>
> 2012-09-17 16:00:29,667 INFO main ChukwaAgent - starting adaptors listed
> in /opt/chukwa/etc/chukwa/initial_adaptors****
>
> 2012-09-17 16:00:29,675 INFO main AgentControlSocketListener -
> AgentControlSocketListerner ask for port: 9093****
>
> 2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - socket
> bound to 9093****
>
> 2012-09-17 16:00:29,675 INFO main ChukwaAgent - control socket started on
> port 9093****
>
> 2012-09-17 16:00:29,713 INFO main log - Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via

RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

Kumar, Suresh 2012-09-18, 17:45

I had checked these before posting, the problem was in the “initial_adaptors” file, I had copied

“add FileTailer FooData /tmp/foo 0” from the Chukwa agent documentation, it didn’t work. I had

to change to “add filetailer.FileTailingAdaptor FooData /tmp/foo 0” to make it work. I found the

exception in “agent.log” and fixed the problem.

However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any clues?

Thanks,

Suresh

2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw

java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING

   at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)

   at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)

   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)

   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)

   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)

   at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)

   at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)

   at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)

   at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)

   at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)

   at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)

   at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)

   at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)

   at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)

   at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)

   at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

   at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)

   at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)

   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)

   at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

   at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

   at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)

   at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

   at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

   at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)

   at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)

   at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)

   at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)

   at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)

   at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)

   at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

   at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

   at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)

   at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

   at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

   at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)

   at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)

   at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)

   at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)

   at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

   at org.mortbay.jetty.Server.handle(Server.java:326)

   at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

   at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) 2012-09-17 18:04:25,700 ERROR 2124414527@qtp-1510130526-7-EventThread ClientCnxn - Error while calling watcher

java.lang.NullPointerException: ZK is null

   at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:338)

   at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:275)

   at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)

   at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)

   at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)

   at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)

   at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

   at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)

   at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

2012-09-17 18:04:28,694 INFO 2124414527@qtp-1510130526-7 RecoverableZooKeeper - The identifier of this process is 18719@localhost

2012-09-17 18:05:14,141 INFO 1790591505@qtp-1510130526-5 /hicc - jsp: init

2012-09-17 19:45:27,547 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf is /opt/chukwa/etc/chukwa

2012-09-17 19:45:28,125 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf is /opt/chukwa/etc/chukwa

2012-09-17 19:45:28,421 ERROR 1983406566@qtp-1510130526-10 WidgetB

RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

Shao, Saisai 2012-09-19, 01:06

Hi Suresh,
I met this exception before.
ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException
   at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
   at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
   at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
   at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
   at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
   at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
   at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
   at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
   at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
   at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)
I think this is a bug in HICC while reading descriptors from HDFS, at that time I delete this file (/chukwa/hicc/widgets/disabled) on hdfs and restart HICC, you can try it.
I’m sorry for the other Zookeeper exceptions, also In my environment, HICC only take several minutes to load.

Thanks
Saisai
From: Kumar, Suresh [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, September 19, 2012 1:45 AM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

I had checked these before posting, the problem was in the “initial_adaptors” file, I had copied
“add FileTailer FooData /tmp/foo 0” from the Chukwa agent documentation, it didn’t work. I had
to change to “add filetailer.FileTailingAdaptor FooData /tmp/foo 0” to make it work. I found the
exception in “agent.log” and fixed the problem.

However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any clues?

Thanks,
Suresh

2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
   at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)
   at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)
   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)
   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)
   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)
   at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)
   at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)
   at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)
   at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)
   at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)
   at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)
   at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)
   at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)
   at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)
   at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
   at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
   at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
   at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
   at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
   at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
   at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
   at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
   at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
   at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
   at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
   at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
   at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
   at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
   at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
   at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
   at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
   at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
   at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
   at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
   at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
   at org.mortbay.jetty.Server.handle(Server.java:326)
   at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
   at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnect

RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

Kumar, Suresh 2012-09-19, 20:26

Thanks Saisai, I removed the disabled file, still see the exception.

What version of chukwa are you running? What version of HBase are you running?

Looking at the exception in hicc.out, looks like it is a Zookeeper exception which probably

originates from HBase.

2012-09-19 14:57:17,139 INFO 2124414527@qtp-1510130526-7-SendThread(localhost:2181) ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x139df9f4b070005, negotiated timeout = 40000

2012-09-19 14:57:19,142 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw

java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING



Suresh

From: Shao, Saisai [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, September 18, 2012 6:06 PM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

Hi Suresh,

I met this exception before.

ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException

   at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)

   at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)

   at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)

   at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)

   at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)

   at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)

   at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)

   at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)

   at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)

   at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)

I think this is a bug in HICC while reading descriptors from HDFS, at that time I delete this file (/chukwa/hicc/widgets/disabled) on hdfs and restart HICC, you can try it.

I’m sorry for the other Zookeeper exceptions, also In my environment, HICC only take several minutes to load.

Thanks

Saisai

From: Kumar, Suresh [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, September 19, 2012 1:45 AM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

I had checked these before posting, the problem was in the “initial_adaptors” file, I had copied

“add FileTailer FooData /tmp/foo 0” from the Chukwa agent documentation, it didn’t work. I had

to change to “add filetailer.FileTailingAdaptor FooData /tmp/foo 0” to make it work. I found the

exception in “agent.log” and fixed the problem.

However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any clues?

Thanks,

Suresh

2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw

java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING

   at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)

   at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)

   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)

   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)

   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)

   at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)

   at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)

   at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)

   at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)

   at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)

   at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)

   at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)

   at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)

   at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)

   at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)

   at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

   at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)

   at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)

   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)

   at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

   at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

   at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)

   at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

   at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

   at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)

   at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)

   at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)

   at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)

   at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)

   at javax.servlet.http.HttpServlet.serv

RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

Shao, Saisai 2012-09-20, 00:10

Hi Suresh,
Removing disabled file can only fix this exception(WidgetBean - java.lang.NullPointerException).
For the zookeeper problem, I agreed with you that it is originated from HBase, but I have no clues about this exception. Did you config HBase related environment variables in chukwa-env.sh, or did you copy hbase-site.xml to etc/chukwa folder.
My chukwa version 0.6.0, currently trunk version, and my HBase version is 0.94.0. Sorry for not helping you with the zookeeper problems.

Thanks
Saisai

From: Kumar, Suresh [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 20, 2012 4:27 AM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

Thanks Saisai, I removed the disabled file, still see the exception.

What version of chukwa are you running? What version of HBase are you running?
Looking at the exception in hicc.out, looks like it is a Zookeeper exception which probably
originates from HBase.

2012-09-19 14:57:17,139 INFO 2124414527@qtp-1510130526-7-SendThread(localhost:2181) ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x139df9f4b070005, negotiated timeout = 40000
2012-09-19 14:57:19,142 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
Suresh
From: Shao, Saisai [mailto:[EMAIL PROTECTED]]<mailto:[mailto:[EMAIL PROTECTED]]>
Sent: Tuesday, September 18, 2012 6:06 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

Hi Suresh,
I met this exception before.
ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException
   at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
   at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
   at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
   at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
   at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
   at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
   at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
   at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
   at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
   at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)
I think this is a bug in HICC while reading descriptors from HDFS, at that time I delete this file (/chukwa/hicc/widgets/disabled) on hdfs and restart HICC, you can try it.
I’m sorry for the other Zookeeper exceptions, also In my environment, HICC only take several minutes to load.

Thanks
Saisai
From: Kumar, Suresh [mailto:[EMAIL PROTECTED]]<mailto:[mailto:[EMAIL PROTECTED]]>
Sent: Wednesday, September 19, 2012 1:45 AM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

I had checked these before posting, the problem was in the “initial_adaptors” file, I had copied
“add FileTailer FooData /tmp/foo 0” from the Chukwa agent documentation, it didn’t work. I had
to change to “add filetailer.FileTailingAdaptor FooData /tmp/foo 0” to make it work. I found the
exception in “agent.log” and fixed the problem.

However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any clues?

Thanks,
Suresh

2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
   at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)
   at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)
   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)
   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)
   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)
   at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)
   at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)
   at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)
   at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)
   at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)
   at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)
   at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)
   at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)
   at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)
   at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
   at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
   at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
   at javax.servlet.http.HttpServlet

RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

Kumar, Suresh 2012-09-21, 22:17

Hi Saisai:

I have the hbase-site.xml in the etc/chukwa folder.

I upgraded to 0.6.0, the exception is gone,

Thanks,

Suresh

From: Shao, Saisai [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, September 19, 2012 5:10 PM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

Hi Suresh,

Removing disabled file can only fix this exception(WidgetBean - java.lang.NullPointerException).

For the zookeeper problem, I agreed with you that it is originated from HBase, but I have no clues about this exception. Did you config HBase related environment variables in chukwa-env.sh, or did you copy hbase-site.xml to etc/chukwa folder.

My chukwa version 0.6.0, currently trunk version, and my HBase version is 0.94.0. Sorry for not helping you with the zookeeper problems.

Thanks

Saisai

From: Kumar, Suresh [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 20, 2012 4:27 AM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

Thanks Saisai, I removed the disabled file, still see the exception.

What version of chukwa are you running? What version of HBase are you running?

Looking at the exception in hicc.out, looks like it is a Zookeeper exception which probably

originates from HBase.

2012-09-19 14:57:17,139 INFO 2124414527@qtp-1510130526-7-SendThread(localhost:2181) ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x139df9f4b070005, negotiated timeout = 40000

2012-09-19 14:57:19,142 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw

java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING



Suresh

From: Shao, Saisai [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, September 18, 2012 6:06 PM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

Hi Suresh,

I met this exception before.

ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException

   at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)

   at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)

   at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)

   at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)

   at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)

   at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)

   at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)

   at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)

   at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)

   at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)

I think this is a bug in HICC while reading descriptors from HDFS, at that time I delete this file (/chukwa/hicc/widgets/disabled) on hdfs and restart HICC, you can try it.

I’m sorry for the other Zookeeper exceptions, also In my environment, HICC only take several minutes to load.

Thanks

Saisai

From: Kumar, Suresh [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, September 19, 2012 1:45 AM
To: [EMAIL PROTECTED]
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out

I had checked these before posting, the problem was in the “initial_adaptors” file, I had copied

“add FileTailer FooData /tmp/foo 0” from the Chukwa agent documentation, it didn’t work. I had

to change to “add filetailer.FileTailingAdaptor FooData /tmp/foo 0” to make it work. I found the

exception in “agent.log” and fixed the problem.

However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any clues?

Thanks,

Suresh

2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw

java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING

   at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)

   at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)

   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)

   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)

   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)

   at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)

   at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)

   at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)

   at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)

   at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)

   at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)

   at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)

   at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)

   at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)

   at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)

   at javax.servlet.http.