Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Region server instability with large number of deletes

Mark Greene 2012-12-12, 19:16
Copy link to this message
Re: Region server instability with large number of deletes
That is fixed in 0.94 (can't find the exact issue now, though)

Before the fix in 0.94 the region server would skip deleted columns one by one in order to advance to the next row.
Now it seeks ahead, instead.

In 0.92 I think your only option is to follow your batch delete with a major compaction.
-- Lars

 From: Mark Greene <[EMAIL PROTECTED]>
Sent: Wednesday, December 12, 2012 11:16 AM
Subject: Region server instability with large number of deletes
Hi All,

I'm running HBase 0.92.1-cdh4.1.2.

I'm seeing a region server get into a state where they constantly emit
LeaseExceptions and ClosedChannelException from the IPC handler under load
that is several standard deviations above what we normally see.

One thing that's different about this cluster is we do a lot of deletes
against it. Ultimately all the data in one of our tables is deleted.

When the region server was in this state, the application was doing a lot
of scans, single gets, and then a series of batch deletes. While this cycle
was churning at a higher than average rate then what I normally see, it is
being done in a single thread. There are of course other processes reading
and writing to the cluster while this was going on but the observed load
from those were nothing out of the norm.

So I'm wondering if the large number of deletes via batch is a usage
profile that isn't optimized for hbase?

Other info:
-We just migrated to cdh4.1.2 a few days ago
-Currently do not have major compactions running
-Saw the same behavior in cdh3
-CPU and iowait were low during the time I observed these errors happening.

Thanks in advance,