Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Lots of SocketTimeoutException for gets and puts since HBase 0.92.1


Copy link to this message
-
Lots of SocketTimeoutException for gets and puts since HBase 0.92.1
Guillaume Perrot 2012-11-15, 13:21
Hi all, we just upgraded our HBase cluster from 0.90.3 to 0.92.1 and now we
have a lot of warnings like these in our clients:

2012-11-15 01:31:57,734 | WARN  | <our thread> |
HConnectionManager$HConnectionImplementation
| Failed all from region=<our_table>,0d9750f9e22628e94dd33a78292d62
01,1346224022442.a42b483bb10fbaea70f8616e7f06899c., hostname=<our_host>,
port=60020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException:
Call to <our_host>:60020 failed on socket timeout exception:
java.net.SocketTimeoutException:
60000 millis timeout while waiting for channel to be ready for read. ch :
ja\
va.nio.channels.SocketChannel[connected local=/<some_host>:12492
remote=<our_host>]
        at java.util.concurrent.FutureTask$Sync.innerGet(
FutureTask.java:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.apache.hadoop.hbase.client.HConnectionManager$
HConnectionImplementation.processBatchCallback(HConnectionManager.java:1557)
        at org.apache.hadoop.hbase.client.HConnectionManager$
HConnectionImplementation.processBatch(HConnectionManager.java:1409)
        at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:746)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:715)
        at org.apache.hadoop.hbase.client.HTablePool$
PooledHTable.get(HTablePool.java:371)
        at <our_client_code>
Caused by: java.net.SocketTimeoutException: Call to <our_host>:60020 failed
on socket timeout exception: java.net.SocketTimeoutException: 60000 millis
timeout while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[\
connected local=<some_host>:12492 remote=<our_host>:60020]
        at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(
HBaseClient.java:949)
        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.
java:922)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.
invoke(WritableRpcEngine.java:150)
        at $Proxy7.multi(Unknown Source)
        at org.apache.hadoop.hbase.client.HConnectionManager$
HConnectionImplementation$3$1.call(HConnectionManager.java:1386)
        at org.apache.hadoop.hbase.client.HConnectionManager$
HConnectionImplementation$3$1.call(HConnectionManager.java:1384)
        at org.apache.hadoop.hbase.client.HConnectionManager$
HConnectionImplementation.getRegionServerWithoutRetries(
HConnectionManager.java:1365)
        at org.apache.hadoop.hbase.client.HConnectionManager$
HConnectionImplementation$3.call(HConnectionManager.java:1383)
        at org.apache.hadoop.hbase.client.HConnectionManager$
HConnectionImplementation$3.call(HConnectionManager.java:1381)

        at java.util.concurrent.FutureTask$Sync.innerRun(
FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

We have that on puts too (puts seems to use a lower value for the socket
timeout):

2012-11-14 20:44:55,320 | WARN  | <our_thread> |
HConnectionManager$HConnectionImplementation | Failed all from
region=<our_table>,,1346224022442.e90f2b7680df46d93d8ecd13eee08265.,
hostname=<our_server>, port=60020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException:
20000 millis timeout while waiting for channel to be ready for connect. ch
: java.nio.channels.SocketChannel[connection-pending
remote=<our_server>:60020]
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1557)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409)
    at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:943)
    at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:820)
    at org.apache.hadoop.hbase.client.HTable.put(HTable.java:803)
    at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:402)
    at <our_client_code>
Caused by: java.net.SocketTimeoutException: 20000 millis timeout while
waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending
remote=<our_server>:60020]
    at org.apache.hadoop.net
.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
    at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupConnection(HBaseClient.java:328)
    at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:362)
    at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1045)
    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897)
    at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
    at $Proxy7.multi(Unknown Source)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1386)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1384)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1365)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1383)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1381)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.T
+
Stack 2012-11-16, 00:56
+
Vincent Barat 2012-11-16, 16:14
+
Ted Yu 2012-11-16, 17:13
+
Vincent Barat 2012-11-16, 17:20
+
Vincent Barat 2012-11-16, 19:55