Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> issue with thrift server pile up.


Copy link to this message
-
Re: issue with thrift server pile up.
On Thu, Apr 21, 2011 at 12:21 AM, Stack <[EMAIL PROTECTED]> wrote:
> On Wed, Apr 20, 2011 at 3:45 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>> Hello -- we have an issue that looks like this.   We have php app
>> front end and thrift servers that live on seperate boxes away from
>> HBASE cluster.  Everytime we do compaction on one of our 8 RS servers,
>> we cause a thread pile up on Thrift servers that delay _all_ queries
>> to HBASE.
>
> What kind of compaction.  You are compacting a single region or all
> regions on this box?  Minor or major compaction?
>
Minor compaction for one (single) region on a regionserver.
> What kind of hardware is this?
16G Ram, 8 Core box, 6 disks, CPU user/system/io is slow at 5% or less.

>
 >When compaction runs on this machine,
> it is responsible for upping latency against regions hosted on this
> box alone or on all boxes; i.e. the compaction load is such that hdfs
> on all the cluster slows?
>
>
HDFS on all boxes remain fast, as well as the box doing compaction,
However all queries (scans, gets) to the same region server get slow.
>
>> Our usual level of threads on Thrift is around 250 threads,
>> but when we do a single compaction on a single region server, the
>> Thrift threads jump to about 5,000, which makes HBASE to be slow for
>> all regions , not just the one that compacts.
>
> Can you give us snippets from logs of the regionserver that is
> compacting over the period where the thrift servers backup?
>

Yes:

2011-04-21 00:39:53,342 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Flush requested on
service_message,1_60443424997912576,1303351424896.ccdb0fe2e1976495836e926c57a7a36d.
2011-04-21 00:39:53,342 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush
for region service_message,1_60443424997912576,1303351424896.ccdb0fe2e1976495836e926c57a7a36d..
Current region memstore size 256.0m
2011-04-21 00:39:53,342 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting,
commencing flushing stores
2011-04-21 00:39:59,938 INFO
org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at
hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/.tmp/7579206879034321398
to hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/common/3415152704800366912
2011-04-21 00:40:03,123 INFO
org.apache.hadoop.hbase.regionserver.Store: Added
hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/common/3415152704800366912,
entries=866486, sequenceid=1934982293, memsize=212.7m, filesize=65.3m
to service_message,1_60443424997912576,1303351424896.ccdb0fe2e1976495836e926c57a7a36d.
2011-04-21 00:40:05,611 INFO
org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at
hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/.tmp/4177410838492147388
to hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/link/6676648044671628059
2011-04-21 00:40:06,420 INFO
org.apache.hadoop.hbase.regionserver.Store: Added
hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/link/6676648044671628059,
entries=25296, sequenceid=1934982293, memsize=6.8m, filesize=2.5m to
service_message,1_60443424997912576,1303351424896.ccdb0fe2e1976495836e926c57a7a36d.
2011-04-21 00:40:07,519 INFO
org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at
hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/.tmp/2030906487055196702
to hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/photo/7887239631758414992
2011-04-21 00:40:09,520 INFO
org.apache.hadoop.hbase.regionserver.Store: Added
hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/photo/7887239631758414992,
entries=10896, sequenceid=1934982293, memsize=3.4m, filesize=1.5m to
service_message,1_60443424997912576,1303351424896.ccdb0fe2e1976495836e926c57a7a36d.
2011-04-21 00:40:11,344 INFO
org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at
hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/.tmp/2825213906012468466
to hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/place/8085991053959255054
2011-04-21 00:40:11,540 INFO
org.apache.hadoop.hbase.regionserver.Store: Added
hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/place/8085991053959255054,
entries=1219, sequenceid=1934982293, memsize=334.7k, filesize=122.1k
to service_message,1_60443424997912576,1303351424896.ccdb0fe2e1976495836e926c57a7a36d.
2011-04-21 00:40:12,194 INFO
org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at
hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/.tmp/7899422724259211613
to hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/text/7578566530358624021
2011-04-21 00:40:17,361 INFO
org.apache.hadoop.hbase.regionserver.Store: Added
hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/text/7578566530358624021,
entries=108290, sequenceid=1934982293, memsize=32.5m, filesize=14.0m
to service_message,1_60443424997912576,1303351424896.ccdb0fe2e1976495836e926c57a7a36d.
2011-04-21 00:40:17,519 INFO
org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at
hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/.tmp/7207348071728550157
to hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/video/374590162251960181
2011-04-21 00:40:17,530 INFO
org.apache.hadoop.hbase.regionserver.Store: Added
hdfs://mtag1:9000/hbase/service_message/ccdb0fe2e1976495836e926c57a7a36d/video/374590162251960181,
entries=924, sequenceid=1934982293, memsize=265.2k, filesize=102.5k to
service_message,1_60443424997912576,1303351424896.ccdb0fe2e1976495836e926c57a7a36d.
2011-04-21 00:40:17,533 INFO
org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush
of ~256.0m for region
service_message,1_60443424997912576,1303351424896.ccdb0fe2e1976495836e92
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB