Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> max HStoreFile size


Copy link to this message
-
Re: max HStoreFile size
I set the compaction policy to constant size and still, when compacting
bulk loaded regions, it splits the regions (the region's size is much
smaller than max file size but I do use compression...)
On Jan 23, 2014 12:11 PM, "Samir Ahmic" <[EMAIL PROTECTED]> wrote:

> Hi Amit,
>
> Yes. You can set split policy per table. Here is relevant part of hbase
> book:
>
> http://hbase.apache.org/book/regions.arch.html
>
> The policy can set globally through the HBaseConfiguration used or on a per
> table basis:
>
> HTableDescriptor myHtd = ...;
> myHtd.setValue(HTableDescriptor.SPLIT_POLICY,
> MyCustomSplitPolicy.class.getName());
>
> Cheers
>
>
>
> On Thu, Jan 23, 2014 at 9:34 AM, Amit Sela <[EMAIL PROTECTED]> wrote:
>
> > So I think my problem is that from 0.94 the default split policy
> > is: IncreasingToUpperBoundRegionSplitPolicy and
> > not ConstantSizeRegionSplitPolicy.
> >
> > Can I set split policy per table ?
> >
> > Still don't know if hbase.hregion.max.filesize relates to compressed or
> > uncompressed ?
> >
> >
> > On Wed, Jan 22, 2014 at 7:25 PM, Amit Sela <[EMAIL PROTECTED]> wrote:
> >
> > > Hi all, I'm using HBase 0.94.12 and in some tables I'm managing
> splitting
> > > and compactions manually.
> > >
> > > I was wondering if hbase.hregion.max.filesize relates to compressed or
> > > uncompressed file size.
> > > If I'm using compression, and the file size <
> hbase.hregion.max.filesize
> > > but uncompressed it's bigger, than when executing major compaction on
> the
> > > region, it splits.
> > >
> > > Should it be like that ? more important, the recommendation of regions
> of
> > > 1GB is for compressed or uncompressed StoreFile size?
> > >
> > > Since I'm using bulk load, I get about 3 StoreFiles loaded into each CF
> > of
> > > every new region, I executed region compaction to unite them as 1 file
> > (and
> > > then got the unwanted splits) - If I'm never updating this data, do I
> > gain
> > > something from uniting the files ?
> > > Could I manage ~500MB of compressed (GZ - decompresses to about 7.5GB)
> > > with 10GB RAM RegionServers ?
> > >
> > > Thanks,
> > >
> > > Amit.
> > >
> > >
> > >
> > >
> > >
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB