Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Blur >> mail # user >> Block-Cache and usage


Copy link to this message
-
Re: Block-Cache and usage
On Thu, Mar 27, 2014 at 8:27 AM, Ravikumar Govindarajan <
[EMAIL PROTECTED]> wrote:
It's not exactly a byte[].  Default configuration is a UnsafeCacheValue
wrapped by a DetachableCacheValue.  The UnsafeCacheValue allocates off heap
memory.  The DetachableCacheValue handles when the CacheValue is still in
use by some reference but needs to be evicted from the block cache.  So
when this occurs it copies the data from the UnsafeCacheValue (off heap) to
a ByteArrayCacheValue (on heap) and then the JVM handles things from
there.  However this situation doesn't happen very often.

https://git-wip-us.apache.org/repos/asf?p=incubator-blur.git;a=blob;f=blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/UnsafeCacheValue.java;h=6f230fd94f37de994841f0df96086906bf7ea1ae;hb=9de8a93aef77aa6bd8f91e8bfafc647a3a806ed0

https://git-wip-us.apache.org/repos/asf?p=incubator-blur.git;a=blob;f=blur-store/src/main/java/org/apache/blur/store/blockcache_v2/cachevalue/DetachableCacheValue.java;h=959e747b5d857daeee2b35fbb151e7a1581774e6;hb=9de8a93aef77aa6bd8f91e8bfafc647a3a806ed0

Actually we don't.  If you look at this code:

https://git-wip-us.apache.org/repos/asf?p=incubator-blur.git;a=blob;f=blur-store/src/main/java/org/apache/blur/store/blockcache_v2/BaseCache.java;h=4805d40e74759c30ca9ae2016ecd3343eedc6b9e;hb=9de8a93aef77aa6bd8f91e8bfafc647a3a806ed0#l65

You will see that we return the allocated memory to a pool once it has been
evicted from the cache.  The references in the CacheIndexInput doesn't
actually manage the memory, it only holds references to the cache.

Well we never cache 300 MB at once.  Since Lucene is heavy on random
access, it typically accesses small portions of the files rapidly.  And the
cache system caches 8 KB (by default) of file data at a time.  So if a file
has 300 MB present in the cache it is very likely that it is not in
contiguous memory.

Hope this helps.

Aaron