Mark Greene 2012-12-12, 19:16
-Re: Region server instability with large number of deletes
lars hofhansl 2012-12-13, 01:14
That is fixed in 0.94 (can't find the exact issue now, though)
Before the fix in 0.94 the region server would skip deleted columns one by one in order to advance to the next row.
Now it seeks ahead, instead.
In 0.92 I think your only option is to follow your batch delete with a major compaction.
From: Mark Greene <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Wednesday, December 12, 2012 11:16 AM
Subject: Region server instability with large number of deletes
I'm running HBase 0.92.1-cdh4.1.2.
I'm seeing a region server get into a state where they constantly emit
LeaseExceptions and ClosedChannelException from the IPC handler under load
that is several standard deviations above what we normally see.
One thing that's different about this cluster is we do a lot of deletes
against it. Ultimately all the data in one of our tables is deleted.
When the region server was in this state, the application was doing a lot
of scans, single gets, and then a series of batch deletes. While this cycle
was churning at a higher than average rate then what I normally see, it is
being done in a single thread. There are of course other processes reading
and writing to the cluster while this was going on but the observed load
from those were nothing out of the norm.
So I'm wondering if the large number of deletes via batch is a usage
profile that isn't optimized for hbase?
-We just migrated to cdh4.1.2 a few days ago
-Currently do not have major compactions running
-Saw the same behavior in cdh3
-CPU and iowait were low during the time I observed these errors happening.
Thanks in advance,