Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Coprocessor tests under busy insertions


Copy link to this message
-
Coprocessor tests under busy insertions
Hi, hbase users.

now, I am testing coprocessors to create secondary indexes in background.
coprocessors itself is packaged in base 0.92.1 I am using.

the scenario I want to describe is this one.

the main table is 'blog' which is having a field named 'userId'.
from this field I want to create secondary index to map 'userId' and it's 'url'.

I put RegionObserver implementations in my secondary index creator.

the situation I got from hbase is this log.

------------
12/08/13 10:37:08 WARN client.HConnectionManager$HConnectionImplementation: Failed all from region=blog,user6447991910946051755,1344821177585.7d4cbd4a9817ab7cb5c6219498d854a4., hostname=search-ddm-test5, port=60020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Call to search-ddm-test5/xx.xx.xx.xx:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/xx.xx.xx.xx:53733 remote=search-ddm-test5/xx.xx.xx.xx:60020]
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1557)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:943)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:820)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:795)
at com.yahoo.ycsb.db.HBaseClient.update(HBaseClient.java:321)
at com.yahoo.ycsb.DBWrapper.update(DBWrapper.java:126)
at com.yahoo.ycsb.workloads.CoreWorkload.doTransactionUpdate(CoreWorkload.java:628)
at com.yahoo.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:483)
at com.yahoo.ycsb.ClientThread.run(Client.java:233)
Caused by: java.net.SocketTimeoutException: Call to search-ddm-test5/10.11.239.220:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/xx.xx.xx.xx:53733 remote=search-ddm-test5/xx.xx.xx.xx:60020]
at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:949)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:922)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy5.multi(Unknown Source)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1386)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1384)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1365)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1383)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1381)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/xx.xx.xx.xx:53733 remote=search-ddm-test5/xx.xx.xx.xx:60020]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:311)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:571)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
 200 sec: 237725 operations; 0 current ops/sec;  
 210 sec: 237725 operations; 0 current ops/sec;  
 220 sec: 237725 operations; 0 current ops/sec;
….
….

 560 sec: 237725 operations; 0 current ops/sec;  
 570 sec: 237725 operations; 0 current ops/sec;  
 580 sec: 237725 operations; 0 current ops/sec;  
 590 sec: 237725 operations; 0 current ops/sec;  
 600 sec: 237725 operations; 0 current ops/sec;  
 610 sec: 237725 operations; 0 current ops/sec;  
 620 sec: 237725 operations; 0 current ops/sec;  
 630 sec: 237725 operations; 0 current ops/sec;  
 640 sec: 237725 operations; 0 current ops/sec;  
 650 sec: 237725 operations; 0 current ops/sec;  
 660 sec: 237725 operations; 0 current ops/sec;  
 670 sec: 237725 operations; 0 current ops/sec;  
 680 sec: 237725 operations; 0 current ops/sec;  
 690 sec: 237725 operations; 0 current ops/sec;  
 700 sec: 237725 operations; 0 current ops/sec;  
 710 sec: 237725 operations; 0 current ops/sec;  
 720 sec: 237725 operations; 0 current ops/sec;  
 730 sec: 237725 operations; 0 current ops/sec;  
 740 sec: 237725 operations; 0 current ops/sec;  
 750 sec: 237725 operations; 0 current ops/sec;  
 760 sec: 237725 operations; 0 current ops/sec;

there was several normal insertions about 10000 TPS.
but, after this operations, no more insertions, just warnings from hbase.

in my implementations,
I didn't use HTablePool, Counter, only thing I used is just to put
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB