Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> RS unresponsive after series of deletes


+
Ted Tuttle 2012-06-13, 19:09
+
Stack 2012-06-14, 17:38
Copy link to this message
-
RE: RS unresponsive after series of deletes
> What kind of a delete are you doing?  

A mixture of row and cell deletes.  Interestingly, the first 19
(successful) deletes were row deletes.  The client got hung up while
submitting its first batch of cell deletes.  However, I think the
cell/row distinction is a red herring as we've experienced this behavior
at least once with batches of exclusively row deletes.

> When you say 19 deletes, each of these is a batch delete?

Each of the 19 deletes is a call to HTable.delete(List<Delete>).  I
estimated there where about 144 Deletes in each batch.  In the cell
delete that failed, I estimate about 1000 column qualifiers per row for
a total of about 144k cells per batch.
  
> Could it be that a batch is doing a bunch at the one time and taking a
long time to complete?  

In order to issue the cell delete we scan each row's column keys for
matches to in-memory set of domain objects.  The code to construct the
delete is completing quickly.

I should add that most of our deletes are very fast. But on 3 occasions
thus far, they exceed 10min allotted by retry logic in client.

> Try making smaller batches?  Want to try thread dumping it when it
goes unresponsive?

I will try to reproduce w/ test harness.
  
> Do you have gc logging enabled?  Anything in the .out file at this
time when we are using CPU?

I don't see any GC related operations over 10s.  Here is log from time
of first failure to 20min after: http://pastebin.com/AUaULHcD

-Ted
+
Ted Tuttle 2012-06-18, 22:08
+
Jean-Daniel Cryans 2012-06-18, 22:17
+
Ted Tuttle 2012-06-20, 20:49
+
Jean-Daniel Cryans 2012-06-20, 21:32
+
Ted Tuttle 2012-06-21, 00:07
+
Ted Yu 2012-06-21, 03:52
+
Ted Yu 2012-06-21, 04:33
+
Ted Tuttle 2012-06-21, 04:54
+
Ted Yu 2012-06-21, 05:13
+
Ted Tuttle 2012-06-21, 14:02
+
Ted Yu 2012-06-21, 17:32
+
Ted Tuttle 2012-06-21, 18:00
+
Ted Yu 2012-06-21, 18:10
+
Ted Yu 2012-06-21, 14:19
+
Ted Tuttle 2012-06-21, 18:16
+
lars hofhansl 2012-06-23, 02:35
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB