Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - MAX_FILESIZE and hbase.hregion.max.filesize  are both 10Gb


Copy link to this message
-
Re: MAX_FILESIZE and hbase.hregion.max.filesize are both 10Gb
Jean-Marc Spaggiari 2013-07-29, 00:50
"Yes it works, of course." It's not working for me ;) so was not sure.

It's normal to have regions under the half of the MAX_FILESIZE. When a
regions is more than MAX_FILESIZE it's splitted in 2. So one can be more,
and the other one can be less.

I will say, average 5GB will have been a good value, but even 3.6 is still
not so bad.

Have you pre-splitted the regions initially? Is it possible that you have
not-used pre-splitted regions?

You can you Hannibal to have a quick view of what the sizes are

JM

2013/7/28 Vladimir Rodionov <[EMAIL PROTECTED]>

> The final stats:
>
> Total HDFS size - 376GB
> #regions: 109 - avg. region size ~ 3.6GB
>
> Something is wrong here. I expected fewer regions. The regions get split
> at sizes much lower than
> hbase.hregion.max.filesize and/or  MAX_FILESIZE.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Vladimir Rodionov
> Sent: Sunday, July 28, 2013 3:35 PM
> To: [EMAIL PROTECTED]
> Subject: RE: MAX_FILESIZE and hbase.hregion.max.filesize are both 10Gb
>
> Yes it works, of course.
>
> Its in original post - ~ 10gB
>
> <property>
> <name>hbase.hregion.max.filesize</name>
>    <value>10737418240</value>
>    <source>hbase-site.xml</source>
> </property>
>
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Jean-Marc Spaggiari [[EMAIL PROTECTED]]
> Sent: Sunday, July 28, 2013 2:30 PM
> To: [EMAIL PROTECTED]
> Subject: Re: MAX_FILESIZE and hbase.hregion.max.filesize are both 10Gb
>
> Hi Vladimir,
>
> Is this link working for you? http://MASTERURL:60010/conf ? If yes, what
> do
> you have for hbase.hregion.max.filesize? To make sure the property below is
> considerered.
>
> For the table config, did you get it from the webui?
>
> JM
>
> 2013/7/28 Vladimir Rodionov <[EMAIL PROTECTED]>
>
> > but all regions keep getting split at 1Gb
> >
> > I have 71 regions and 70GB of data in 'usertable' despite the fact that:
> >
> > table config is:
> > {NAME => 'usertable', DEFERRED_LOG_FLUSH => 'true', MAX_FILESIZE =>
> > '10000000000', FAMILIES => [{NAME => 'cf', BLOOMFILTER => 'ROWCOL',
> > VERSIONS => '1', COMPRESSION => 'GZ'}]}
> >
> > and hbase-size.xml has the following config:
> >
> > <property>
> >    <name>hbase.hregion.max.filesize</name>
> >    <value>10737418240</value>
> >    <source>hbase-site.xml</source>
> > </property>
> >
> > HBase 0.94.6
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com
> > e-mail: [EMAIL PROTECTED]
> >
> > Confidentiality Notice:  The information contained in this message,
> > including any attachments hereto, may be confidential and is intended to
> be
> > read only by the individual or entity to whom this message is addressed.
> If
> > the reader of this message is not the intended recipient or an agent or
> > designee of the intended recipient, please note that any review, use,
> > disclosure or distribution of this message or its attachments, in any
> form,
> > is strictly prohibited.  If you have received this message in error,
> please
> > immediately notify the sender and/or [EMAIL PROTECTED] and
> > delete or destroy any copy of this message and its attachments.
> >
>
> Confidentiality Notice:  The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited.  If you have received this message in error, please
> immediately notify the sender and/or [EMAIL PROTECTED] and