Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> HBase 0.94.15: writes stalls periodically even under moderate steady load (AWS EC2)


Copy link to this message
-
Re: HBase 0.94.15: writes stalls periodically even under moderate steady load (AWS EC2)
Hi Vladimir,

5k ops/sec over 5 hosts should be entirely reasonable, though EC2 can be a
fickle friend for HBase. Can you run some other task on the host that also
writes data and monitor disk activity with iostat or iotop? Basically, find
some indication if it's the virtual host's IO that's shutting down or if
it's something in HBase itself. If disk IO is looking good, I think I'd
move up the stack and monitor network activity -- are the RegionServers
receiving the write requests?

-n
On Wed, Jan 15, 2014 at 12:33 PM, Vladimir Rodionov <[EMAIL PROTECTED]
> wrote:

> This is something which needs to be definitely solved/fixed/resolved
>
> I am running YCSB benchmark on aws ec2 on a small HBase cluster
>
> 5 (m1.xlarge) as RS
> 1 (m1.xlarge) hbase-master, zookeper
>
> Whirr 0.8.2 (with many hacks) is used to provision HBase.
>
> I am running 1 ycsb client (100% insert ops) throttled at 5K ops:
>
> ./bin/ycsb load hbase -P workloads/load20m -p columnfamily=family -s
> -threads 10 -target 5000
>
> OUTPUT:
>
> 1120 sec: 5602339 operations; 4999.7 current ops/sec; [INSERT
> AverageLatency(us)=225.53]
>  1130 sec: 5652117 operations; 4969.35 current ops/sec; [INSERT
> AverageLatency(us)=203.31]
>  1140 sec: 5665210 operations; 1309.04 current ops/sec; [INSERT
> AverageLatency(us)=17.13]
>  1150 sec: 5665210 operations; 0 current ops/sec;
>  1160 sec: 5665210 operations; 0 current ops/sec;
>  1170 sec: 5665210 operations; 0 current ops/sec;
>  1180 sec: 5665210 operations; 0 current ops/sec;
>  1190 sec: 5665210 operations; 0 current ops/sec;
> 2014-01-15 15:19:34,139 Thread-2 WARN
>  [HConnectionManager$HConnectionImplementation] Failed all from
> region=usertable,user6039,1389811852201.40518862106856d23b883e5d543d0b89.,
> hostname=ip-10-45-174-120.ec2.internal, port=60020
> java.util.concurrent.ExecutionException: java.net.SocketTimeoutException:
> Call to ip-10-45-174-120.ec2.internal/10.45.174.120:60020 failed on
> socket timeout exception: java.net.SocketTimeoutException: 60000 millis
> timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.180.211.173:42466remote=ip-10-45-174-120.ec2.internal/
> 10.45.174.120:60020]
>         at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>         at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1708)
>         at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1560)
>         at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:994)
>         at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:850)
>         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:826)
>         at com.yahoo.ycsb.db.HBaseClient.update(HBaseClient.java:328)
>         at com.yahoo.ycsb.db.HBaseClient.insert(HBaseClient.java:357)
>         at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148)
>         at
> com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461)
>         at com.yahoo.ycsb.ClientThread.run(Client.java:269)
> Caused by: java.net.SocketTimeoutException: Call to
> ip-10-45-174-120.ec2.internal/10.45.174.120:60020 failed on socket
> timeout exception: java.net.SocketTimeoutException: 60000 millis timeout
> while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.180.211.173:42466remote=ip-10-45-174-120.ec2.internal/
> 10.45.174.120:60020]
>         at
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1043)
>         at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1016)
>         at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87)
>         at com.sun.proxy.$Proxy5.multi(Unknown Source)
>         at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1537)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB