Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Size of KeyValue

Copy link to this message
Re: Size of KeyValue
Even if the blocks aren't being saved into the block cache, aren't they
still getting created in the young gen as they're read from disk?  They
probably get left behind by the young gen collector, but still cause it to
fill up quickly and run more frequent collections.  Every time a collection
runs it has to dig up all the live objects and copy them to a new location.

I can't find much documentation on it, but has anyone tried the
-XX:PretenureSizeThreshold GC flag?  Supposedly you can specify that
objects over a certain number of bytes get allocated directly in the old
gen.  If typical block size is 64KB, then maybe setting this to 32000 will
cause less pollution of the young gen.  Of course, there could be downsides
like greater heap fragmentation and slower allocation.
On Thu, Dec 1, 2011 at 3:39 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> For my scans I have caching disabled. There must be something else
> producing significant amounts of garbage.
> ----- Original Message -----
> From: Dhruba Borthakur <[EMAIL PROTECTED]>
> Cc: lars hofhansl <[EMAIL PROTECTED]>
> Sent: Thursday, December 1, 2011 3:19 PM
> Subject: Re: Size of KeyValue
> Most of our "garbage" is from block cache, not directly from the KVs. Is
> that what you see?
> thanks,
> dhruba
> On Thu, Dec 1, 2011 at 11:06 AM, Stack <[EMAIL PROTECTED]> wrote:
> > On Thu, Dec 1, 2011 at 10:57 AM, lars hofhansl <[EMAIL PROTECTED]>
> > wrote:
> > > To try this out I changed the server side code to keep track of the
> > latest KV rather than the row, and also remove the caching from KV.
> > > The difference for GC and runtime is within the noise. I have to
> > conclude that allocating KVs is just not a big problem compared to other
> > garbage being
> > > produced during scans.
> > >
> > >
> >
> > Ain't all our garbage KVs?  (Though I suppose sometimes its blocks of
> > KVs whether mslab or blocks from hdfs).
> > St.Ack
> >
> --
> Subscribe to my posts at http://www.facebook.com/dhruba