Hi,

I am running hbase 0.94.3 and I am trying to understand the LRU cache stats
here:

LRU Stats: total=428.75 MB, free=4.05 GB, max=4.46 GB, blocks=35961,
accesses=23808283, hits=23098837, hitRatio=97.02%,
cachingAccesses=23135166, cachingHits=23066278,
cachingHitsRatio=99.70%, *evictions=0,
evicted=33211*
*
*
I don't understanding why the number of evictions is 0 - which means that
the number of eviction thread runs is 0, AFAIK. Also, I see that 30K blocks
were evicted - how is that possible ? I have set the hfile evict on close
to the default value - which is false - so blocks should not be evicted
when hfiles are closed. Where are the evicted blocks coming from when the
block is mostly empty and no eviction runs have occured.

Thanks
Varun

Evicted block count stat is gathered in:

  protected long evictBlock(CachedBlock block) {

    map.remove(block.getCacheKey());

    updateSizeMetrics(block, true);

    elements.decrementAndGet();

    stats.evicted();

In EvictionThread#run():

        LruBlockCache cache = this.cache.get();

        if(cache == null) break;

        cache.evict();

where cache is declared:

    private WeakReference<LruBlockCache> cache;

Looks like cache became null (due to heap pressure), leading to 0 evictions
that you saw.
Cheers
On Sat, Jan 26, 2013 at 7:05 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am running hbase 0.94.3 and I am trying to understand the LRU cache stats
> here:
>
> LRU Stats: total=428.75 MB, free=4.05 GB, max=4.46 GB, blocks=35961,
> accesses=23808283, hits=23098837, hitRatio=97.02%,
> cachingAccesses=23135166, cachingHits=23066278,
> cachingHitsRatio=99.70%, *evictions=0,
> evicted=33211*
> *
> *
> I don't understanding why the number of evictions is 0 - which means that
> the number of eviction thread runs is 0, AFAIK. Also, I see that 30K blocks
> were evicted - how is that possible ? I have set the hfile evict on close
> to the default value - which is false - so blocks should not be evicted
> when hfiles are closed. Where are the evicted blocks coming from when the
> block is mostly empty and no eviction runs have occured.
>
> Thanks
> Varun
>

Since i am using only 10 % of allocated cache, I think EvictionThread never
ran - hence, I see the value 0. What's mysterious is who is calling
evictBlock then, because there are non zero evicted blocks ?

On Sat, Jan 26, 2013 at 9:24 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Evicted block count stat is gathered in:
>
>   protected long evictBlock(CachedBlock block) {
>
>     map.remove(block.getCacheKey());
>
>     updateSizeMetrics(block, true);
>
>     elements.decrementAndGet();
>
>     stats.evicted();
>
> In EvictionThread#run():
>
>         LruBlockCache cache = this.cache.get();
>
>         if(cache == null) break;
>
>         cache.evict();
>
> where cache is declared:
>
>     private WeakReference<LruBlockCache> cache;
>
> Looks like cache became null (due to heap pressure), leading to 0 evictions
> that you saw.
>
>
> Cheers
>
>
> On Sat, Jan 26, 2013 at 7:05 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > I am running hbase 0.94.3 and I am trying to understand the LRU cache
> stats
> > here:
> >
> > LRU Stats: total=428.75 MB, free=4.05 GB, max=4.46 GB, blocks=35961,
> > accesses=23808283, hits=23098837, hitRatio=97.02%,
> > cachingAccesses=23135166, cachingHits=23066278,
> > cachingHitsRatio=99.70%, *evictions=0,
> > evicted=33211*
> > *
> > *
> > I don't understanding why the number of evictions is 0 - which means that
> > the number of eviction thread runs is 0, AFAIK. Also, I see that 30K
> blocks
> > were evicted - how is that possible ? I have set the hfile evict on close
> > to the default value - which is false - so blocks should not be evicted
> > when hfiles are closed. Where are the evicted blocks coming from when the
> > block is mostly empty and no eviction runs have occured.
> >
> > Thanks
> > Varun
> >
>

IIRC when a file closes it will evict its own blocks since they won't
be used after that.

J-D

On Sun, Jan 27, 2013 at 1:04 AM, Varun Sharma <[EMAIL PROTECTED]> wrote:
> Since i am using only 10 % of allocated cache, I think EvictionThread never
> ran - hence, I see the value 0. What's mysterious is who is calling
> evictBlock then, because there are non zero evicted blocks ?
>
> On Sat, Jan 26, 2013 at 9:24 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> Evicted block count stat is gathered in:
>>
>>   protected long evictBlock(CachedBlock block) {
>>
>>     map.remove(block.getCacheKey());
>>
>>     updateSizeMetrics(block, true);
>>
>>     elements.decrementAndGet();
>>
>>     stats.evicted();
>>
>> In EvictionThread#run():
>>
>>         LruBlockCache cache = this.cache.get();
>>
>>         if(cache == null) break;
>>
>>         cache.evict();
>>
>> where cache is declared:
>>
>>     private WeakReference<LruBlockCache> cache;
>>
>> Looks like cache became null (due to heap pressure), leading to 0 evictions
>> that you saw.
>>
>>
>> Cheers
>>
>>
>> On Sat, Jan 26, 2013 at 7:05 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:
>>
>> > Hi,
>> >
>> > I am running hbase 0.94.3 and I am trying to understand the LRU cache
>> stats
>> > here:
>> >
>> > LRU Stats: total=428.75 MB, free=4.05 GB, max=4.46 GB, blocks=35961,
>> > accesses=23808283, hits=23098837, hitRatio=97.02%,
>> > cachingAccesses=23135166, cachingHits=23066278,
>> > cachingHitsRatio=99.70%, *evictions=0,
>> > evicted=33211*
>> > *
>> > *
>> > I don't understanding why the number of evictions is 0 - which means that
>> > the number of eviction thread runs is 0, AFAIK. Also, I see that 30K
>> blocks
>> > were evicted - how is that possible ? I have set the hfile evict on close
>> > to the default value - which is false - so blocks should not be evicted
>> > when hfiles are closed. Where are the evicted blocks coming from when the
>> > block is mostly empty and no eviction runs have occured.
>> >
>> > Thanks
>> > Varun
>> >
>>