Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Lots of SocketTimeoutException for gets and puts since HBase 0.92.1


Copy link to this message
-
Lots of SocketTimeoutException for gets and puts since HBase 0.92.1
Hi all, we just upgraded our HBase cluster from 0.90.3 to 0.92.1 and now we
have a lot of warnings like these in our clients:

2012-11-15 01:31:57,734 | WARN  | <our thread> |
HConnectionManager$HConnectionImplementation
| Failed all from region=<our_table>,0d9750f9e22628e94dd33a78292d62
01,1346224022442.a42b483bb10fbaea70f8616e7f06899c., hostname=<our_host>,
port=60020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException:
Call to <our_host>:60020 failed on socket timeout exception:
java.net.SocketTimeoutException:
60000 millis timeout while waiting for channel to be ready for read. ch :
ja\
va.nio.channels.SocketChannel[connected local=/<some_host>:12492
remote=<our_host>]
        at java.util.concurrent.FutureTask$Sync.innerGet(
FutureTask.java:222)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.apache.hadoop.hbase.client.HConnectionManager$
HConnectionImplementation.processBatchCallback(HConnectionManager.java:1557)
        at org.apache.hadoop.hbase.client.HConnectionManager$
HConnectionImplementation.processBatch(HConnectionManager.java:1409)
        at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:746)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:715)
        at org.apache.hadoop.hbase.client.HTablePool$
PooledHTable.get(HTablePool.java:371)
        at <our_client_code>
Caused by: java.net.SocketTimeoutException: Call to <our_host>:60020 failed
on socket timeout exception: java.net.SocketTimeoutException: 60000 millis
timeout while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[\
connected local=<some_host>:12492 remote=<our_host>:60020]
        at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(
HBaseClient.java:949)
        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.
java:922)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.
invoke(WritableRpcEngine.java:150)
        at $Proxy7.multi(Unknown Source)
        at org.apache.hadoop.hbase.client.HConnectionManager$
HConnectionImplementation$3$1.call(HConnectionManager.java:1386)
        at org.apache.hadoop.hbase.client.HConnectionManager$
HConnectionImplementation$3$1.call(HConnectionManager.java:1384)
        at org.apache.hadoop.hbase.client.HConnectionManager$
HConnectionImplementation.getRegionServerWithoutRetries(
HConnectionManager.java:1365)
        at org.apache.hadoop.hbase.client.HConnectionManager$
HConnectionImplementation$3.call(HConnectionManager.java:1383)
        at org.apache.hadoop.hbase.client.HConnectionManager$
HConnectionImplementation$3.call(HConnectionManager.java:1381)

        at java.util.concurrent.FutureTask$Sync.innerRun(
FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

We have that on puts too (puts seems to use a lower value for the socket
timeout):

2012-11-14 20:44:55,320 | WARN  | <our_thread> |
HConnectionManager$HConnectionImplementation | Failed all from
region=<our_table>,,1346224022442.e90f2b7680df46d93d8ecd13eee08265.,
hostname=<our_server>, port=60020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException:
20000 millis timeout while waiting for channel to be ready for connect. ch
: java.nio.channels.SocketChannel[connection-pending
remote=<our_server>:60020]
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1557)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409)
    at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:943)
    at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:820)
    at org.apache.hadoop.hbase.client.HTable.put(HTable.java:803)
    at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:402)
    at <our_client_code>
Caused by: java.net.SocketTimeoutException: 20000 millis timeout while
waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending
remote=<our_server>:60020]
    at org.apache.hadoop.net
.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
    at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupConnection(HBaseClient.java:328)
    at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:362)
    at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1045)
    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897)
    at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
    at $Proxy7.multi(Unknown Source)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1386)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1384)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1365)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1383)
    at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1381)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.T
+
Stack 2012-11-16, 00:56
+
Vincent Barat 2012-11-16, 16:14
+
Ted Yu 2012-11-16, 17:13
+
Vincent Barat 2012-11-16, 17:20
+
Vincent Barat 2012-11-16, 19:55
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB