Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Re: 60000 millis timeout while waiting for channel to be ready for read


Copy link to this message
-
Re: 60000 millis timeout while waiting for channel to be ready for read
Nick Dimiduk 2013-07-30, 01:13
Hi Shapoor,

Moving the conversation to the users list.

Have you solved your issue? Sorry you haven't gotten a response sooner -- I
think everyone is working overtime to get 0.96 released.

I'm assuming each put is independent of the others. You're not putting
100mm times to the same row, are you?  I'm also curious, did you pre-split
your table before starting all of those inserts?

In the log you pasted, it looks like host kcs-testhadoop02 is the one that
times out. Can you reproduce the event and provide for us the RegionServer
logs from the machine that times out around the time of the event. Please
use a pastebin service rather than pasting to the list directly.

Thanks,
Nick

On Tuesday, July 9, 2013, shapoor wrote:

> hello,
>
> i am doing a lot of saves in HBase. like 100,000,000 documents, each 100KB.
> before i start the program there are almost 18 connections after starting
> my
> 2 regions and one master cluster. the connections are zookeeper, hdfs and
> hbase. as the process of saving starts, i have repeatedly more connections
> (i guess for each time flushing) until i reach almost 80 connections and
> that's when the following exeption appears. now HBase manages to save the
> data somehow but it is not effective with so many connections. how do I
> save
> this problem??
>
> regards, shapoor
>
> 13/07/09 13:31:48 WARN client.HConnectionManager$HConnectionImplementation:
> Failed all from
>
> region=table2,doc-id-866604,1373369430484.09001c90b3d2a4c20b56c35bb976ff91.,
> hostname=kcs-testhadoop02, port=60020
> java.util.concurrent.ExecutionException: java.net.SocketTimeoutException:
> Call to kcs-testhadoop02/192.168.111.211:60020 failed on socket timeout
> exception: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/192.168.111.72:55354
> remote=kcs-testhadoop02/192.168.111.211:60020]
>         at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
>         at java.util.concurrent.FutureTask.get(Unknown Source)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1598)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1450)
>         at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:916)
>         at at.knowcenter.backends.HBaseStorage.flush(HBaseStorage.java:324)
>         at at.knowcenter.evaltool.Evaluate.save(Evaluate.java:155)
>         at
> at.knowcenter.evaltool.Evaluate.performSaveEvaluation(Evaluate.java:100)
>         at at.knowcenter.evaltool.Evaluate.evaluate(Evaluate.java:77)
>         at
> at.knowcenter.evaltool.EvaluationTool.execute(EvaluationTool.java:144)
>         at
> at.knowcenter.evaltool.EvaluationTool.main(EvaluationTool.java:199)
> Caused by: java.net.SocketTimeoutException: Call to
> kcs-testhadoop02/192.168.111.211:60020 failed on socket timeout exception:
> java.net.SocketTimeoutException: 60000 millis timeout while waiting for
> channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected
> local=/192.168.111.72:55354 remote=kcs-testhadoop02/192.168.111.211:60020]
>         at
>
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1026)
>         at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:999)
>         at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
>         at com.sun.proxy.$Proxy6.multi(Unknown Source)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1427)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1425)
>         at
>
> org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:215)
>         at
>