Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - 60000 millis timeout while waiting for channel to be ready for read


Copy link to this message
-
Re: 60000 millis timeout while waiting for channel to be ready for read
Ted Yu 2013-07-09, 15:02
Which HBase release are you using ?

On Tue, Jul 9, 2013 at 4:54 AM, shapoor <[EMAIL PROTECTED]> wrote:

> hello,
>
> i am doing a lot of saves in HBase. like 100,000,000 documents, each 100KB.
> before i start the program there are almost 18 connections after starting
> my
> 2 regions and one master cluster. the connections are zookeeper, hdfs and
> hbase. as the process of saving starts, i have repeatedly more connections
> (i guess for each time flushing) until i reach almost 80 connections and
> that's when the following exeption appears. now HBase manages to save the
> data somehow but it is not effective with so many connections. how do I
> save
> this problem??
>
> regards, shapoor
>
> 13/07/09 13:31:48 WARN client.HConnectionManager$HConnectionImplementation:
> Failed all from
>
> region=table2,doc-id-866604,1373369430484.09001c90b3d2a4c20b56c35bb976ff91.,
> hostname=kcs-testhadoop02, port=60020
> java.util.concurrent.ExecutionException: java.net.SocketTimeoutException:
> Call to kcs-testhadoop02/192.168.111.211:60020 failed on socket timeout
> exception: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/192.168.111.72:55354
> remote=kcs-testhadoop02/192.168.111.211:60020]
>         at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
>         at java.util.concurrent.FutureTask.get(Unknown Source)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1598)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1450)
>         at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:916)
>         at at.knowcenter.backends.HBaseStorage.flush(HBaseStorage.java:324)
>         at at.knowcenter.evaltool.Evaluate.save(Evaluate.java:155)
>         at
> at.knowcenter.evaltool.Evaluate.performSaveEvaluation(Evaluate.java:100)
>         at at.knowcenter.evaltool.Evaluate.evaluate(Evaluate.java:77)
>         at
> at.knowcenter.evaltool.EvaluationTool.execute(EvaluationTool.java:144)
>         at
> at.knowcenter.evaltool.EvaluationTool.main(EvaluationTool.java:199)
> Caused by: java.net.SocketTimeoutException: Call to
> kcs-testhadoop02/192.168.111.211:60020 failed on socket timeout exception:
> java.net.SocketTimeoutException: 60000 millis timeout while waiting for
> channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected
> local=/192.168.111.72:55354 remote=kcs-testhadoop02/192.168.111.211:60020]
>         at
>
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1026)
>         at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:999)
>         at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
>         at com.sun.proxy.$Proxy6.multi(Unknown Source)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1427)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1425)
>         at
>
> org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:215)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1434)
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1422)
>         at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>         at java.util.concurrent.FutureTask.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>         at java.lang.Thread.run(Unknown Source)
> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while