Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> optimizing block cache requests + eviction

Viral Bajaria 2013-07-08, 10:04
Ted Yu 2013-07-08, 13:48
Andrew Purtell 2013-07-08, 15:49
Copy link to this message
Re: optimizing block cache requests + eviction
Thanks guys for going through that never-ending email! I will create the
JIRA for block cache eviction and the regionserver assignment command. Ted
already pointed to the JIRA which tries to go a different datanode if the
primary is busy (I will add comments to that one).

To answer Andrews' questions:

- I am using HBase 0.94.4
- I tried taking a stack trace using jstack but after the dump it crashed
the regionserver. I also did not take the dump on the offending
regionserver, rather took it on the regionservers that were making the
block count. I will take a stack trace on the offending server. Is there
any other tool besides jstack ? I don't want to crash my regionserver.
- The HBase clients workload is fairly random and I write to a table every
4-5 seconds. I have varying workloads for different tables. But I do a lot
of batching on the client side and group similar rowkeys together before
doing a GET/PUT. For example: best case I end up doing ~100 puts every
second to a region or in the worst case it's ~5K puts every second. But
again since the workload is fairly random. Currently the clients for the
table which had the most amount of data has been disabled and yet I see the
heavy loads.

To answer Vladimir's points:
- Data access pattern definitely turns out to be uniform over a period of
- I just did a sweep of my code base and found that there are a few places
where Scanner are using block cache. I will disable that and see how it

Andrew Purtell 2013-07-08, 22:53
Viral Bajaria 2013-07-08, 23:28
Jean-Daniel Cryans 2013-07-08, 23:37
Viral Bajaria 2013-07-08, 23:45
Varun Sharma 2013-07-08, 23:50
Viral Bajaria 2013-07-09, 00:20
Jean-Daniel Cryans 2013-07-08, 23:51
Vladimir Rodionov 2013-07-08, 17:45